Anomaly detection by resident device

ABSTRACT

In some implementations, a user device (or resident device) can notify the user of anomalous accessory states. For example, the user device can determine which accessory states and contexts represent normal accessory states in the respective contexts. Similarly to scene suggestions, the user device can analyze historical accessory state data and context data to determine an accessory state pattern that indicates a normal state of an accessory for a given context. The user device can compare the current state and/or context of an accessory to historical accessory state data to determine when the current state of the accessory is abnormal for the current context. If the current accessory state is abnormal for the current context, the user device can present a notification to the user informing the user of the anomalous accessory state.

TECHNICAL FIELD

The disclosure generally relates to controlling remote accessory devicesusing a computing device.

BACKGROUND

Home automation is becoming more and more popular. Starting with homeclothes and dish washing machines years ago to the smart (e.g.,computerized) fixtures, appliances, and accessories we have today, moreand more people are automating their homes. With the increasingavailability of smart accessories and appliances comes more ways tocontrol these smart devices. For example, a software application on auser's mobile device can be configured to control individualaccessories, appliances, and/or fixtures in the user's home or office.However, as accessories get smarter, they also provide a more variedfeature set which makes controlling these devices more and morecomplicated for the user.

SUMMARY

In some implementations, a user device can automatically generate scenesuggestions and/or trigger suggestions. For example, a scene can be acollection of accessories and corresponding accessory states. When thescene is invoked, the user device can cause the accessories associatedwith the scene to assume the corresponding accessory states. The userdevice can automatically determine the accessories and/or correspondingaccessory states for a scene based on historical patterns of userepresented in historical accessory state data. The user device canautomatically determine triggers for automatically invoking scenes basedon historical patterns of use. For example, the user device can analyzehistorical context (e.g., location, time, accessory state, etc.) dataassociated with accessory state transitions to determine triggers forautomatically invoking a scene. The user device can present suggestscenes and/or triggers to the user of the user device.

In some implementations, a resident device can automatically generatescene suggestions and/or trigger suggestions based on historical dataaggregated from multiple user devices. For example, a resident devicecan be a user device that is (e.g., mostly) kept in the homeenvironment. Therefore, the resident device is able to collect accessorystate and/or context data when another user device may not be connectedto the accessories in the home environment. Moreover, the residentdevice can collect and/or aggregate accessory state data, context data,and user device state data from across multiple user devices. Theresident device can automatically determine the accessories and/orcorresponding accessory states for a scene based on historical patternsof use represented in the aggregated historical accessory state data.The user device can automatically determine triggers for automaticallyinvoking scenes based on historical patterns of use. The user device canpresent suggest scenes and/or triggers to the user of the user device.

In some implementations, a user device (or resident device) can notifythe user of anomalous accessory states. For example, the user device candetermine which accessory states and contexts represent normal accessorystates in the respective contexts. Similarly to scene suggestions, theuser device can analyze historical accessory state data and context datato determine an accessory state pattern that indicates a normal state ofan accessory for a given context. The user device can compare thecurrent state and/or context of an accessory to historical accessorystate data to determine when the current state of the accessory isabnormal for the current context. If the current accessory state isabnormal for the current context, the user device can present anotification to the user informing the user of the anomalous accessorystate.

Particular implementations provide at least the following advantages.Scene suggestions can be automatically generated that allow the user torecreate an environment (e.g., accessory states) with a single input.Scene trigger suggestions can be automatically generated that allow theuser to configure the user device to automatically recreate anenvironment (e.g., accessory states) based on a detected context. Theuser device can automatically detect anomalous accessory states andnotify the user of accessory states that might be dangerous or harmfulto the user and/or the user's property.

Details of one or more implementations are set forth in the accompanyingdrawings and the description below. Other features, aspects, andpotential advantages will be apparent from the description and drawings,and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 shows an example home environment.

FIG. 2 shows an example network configuration.

FIG. 3 is a block diagram of an example system for managing accessories.

FIG. 4 is an illustration of an example home environment having varioussmart accessories.

FIG. 5 is an example graphical user interface presented by a homeapplication.

FIG. 6 illustrates an example graphical user interface for presentingsuggested service groups, scenes, and/or scene triggers.

FIG. 7 illustrates an example graphical user interface for presentingdetails about a scene.

FIG. 8 illustrates an example graphical user interface for presentingdetails about a scene trigger.

FIG. 9 illustrates an example graphical user interface for presentinguser approved scene suggestions and triggers.

FIG. 10 illustrates an example graphical user interface for presentinganomalous accessory state data.

FIG. 11 is a flow diagram of an example process for generating scenesuggestions.

FIG. 12 is a flow diagram of an example process for generating scenetrigger suggestions.

FIG. 13 is a flow diagram of an example process for determiningaccessory state anomalies.

FIG. 14 shows a simplified block diagram of an example systemarchitecture for controller.

FIG. 15 shows a simplified block diagram of an example systemarchitecture for accessory.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Example Environment

FIG. 1 shows an example home environment 100. Home environment 100includes a controller 102 that can communicate with various accessorydevices (also referred to as accessories) located in the environment.Controller 102 can include, for example, a desktop computer, laptopcomputer, tablet computer, smart phone, wearable computing device,personal digital assistant, or any other computing device or set ofdevices that is capable of communicating command-and-control messages toaccessories (e.g., as described in U.S. application Ser. No. 14/614,914)and presenting a user interface to allow a user to indicate desiredoperations on the accessories. In some embodiments, controller 102 canbe implemented using multiple discrete devices. For example, there canbe a base station that communicates with accessories and that can beinstalled in a fixed location in environment 100, and one or more mobileremote-control stations (e.g., a handheld or wearable device such as amobile phone, tablet computer, smart watch, eyeglasses, etc.) thatprovide a user interface and communicate with the base station to effectcontrol over accessories. In some embodiments, the base station canfunction as a coordinator or proxy as described below.

Any type of accessory device can be controlled. Examples of accessorydevices include door lock 104, garage door system 106, light fixture108, security camera 110, and thermostat 112. In some instances,controller 102 can communicate directly with an accessory; for instance,controller 102 is shown communicating directly with door lock 104 andgarage door system 106. In other instances, controller 102 cancommunicate via an intermediary. For instance, controller 102 is showncommunicating via a wireless network access point 114 with accessories108, 110, 112 that are on a wireless network provided by access point114. As noted above, in some embodiments, controller 102 can include abase station, and base station functionality can be integrated intoaccess point 114 or into one of the accessories that is to be controlled(e.g., thermostat 112). Another type of intermediary can be coordinator116, which, in addition to operating as a controller, can relay messagesbetween other controllers and accessories. In some embodiments,coordinator 116 can also implement various control logic to automate oroptimize interactions with accessories; examples are described below.

Various communication transports and combinations of transports can beused, and different transports can be used with different devices. Forexample, some wireless transports such as the Bluetooth® Classic orBluetooth® Smart communication protocol and standards promulgated by theBluetooth SIG (referred to herein as “Bluetooth” and “Bluetooth LE”) cansupport direct point-to-point communication between devices within alimited range. Other wireless transports such as a wireless networkcomplying with Wi-Fi® networking standards and protocols promulgated bythe Wi-Fi Alliance (referred to herein as a “Wi-Fi network”) can definea wireless network with a central access point that routescommunications between different devices on the network. Further, whilewireless communication transports are shown, wired transports can alsobe provided for some or all of the accessories. For example, light bulb108 can be connected to access point 114 by a wired connection, andcontroller 102 can communicate with light bulb 108 by sending messageswirelessly to access point 114, which can deliver the messages to lightbulb 108 via the wired connection. As another example, coordinator 116can be connected to access point 114 by a wired connection as shown(this connection can be wireless if desired), and controller 102 cancommunicate with accessories such as light bulb 108 by sending messagesto coordinator 116 via access point 114; coordinator 116 can communicatewith light bulb 108, either via access point 114 or via another channelsuch as a Bluetooth LE channel. Other combinations of wired and wirelesscommunication are also possible.

Further, while one controller 102 is shown, a home environment can havemultiple controller devices. For example, each person who lives in thehome may have his or her own portable device (or devices) that can actas a controller for some or all of accessories 104-112. Differentcontroller devices can be configured to communicate with differentsubsets of the accessories; for example, a child's controller might beblocked from modifying settings on thermostat 112, while a parent'scontroller device is permitted to modify the settings. Such permissionsor privileges can be configured and controlled, for example, usingtechniques described below, and in above-referenced U.S. applicationSer. No. 14/725,891.

In some embodiments, a uniform accessory protocol can facilitatecommunication by a controller 102 with one or more accessories 104-112.The protocol can provide a simple and extensible framework that modelsan accessory as a collection of services, with each service beingdefined as a set of characteristics, each of which has a defined valueat any given time. Various characteristics can represent various aspectsof the accessory's state. For example, in the case of thermostat 112,characteristics can include power (on or off), current temperature, andtarget temperature. In some embodiments, message formats may betransport-dependent while conforming to the same accessory model.Examples of an accessory model based on services and characteristics aredescribed in U.S. application Ser. No. 14/614,914.

The protocol can further define message formats for controller 102 tosend command-and-control messages (requests) to accessory 112 (or otheraccessories) and for accessory 112 to send response messages tocontroller 102. The command-and-control messages can allow controller102 to interrogate the current state of accessory characteristics and insome instances to modify the characteristics (e.g., modifying the powercharacteristic can turn an accessory off or on). Accordingly, any typeof accessory, regardless of function or manufacturer, can be controlledby sending appropriate messages. The format can be the same acrossaccessories. Examples of message formats are described inabove-referenced U.S. application Ser. No. 14/614,914.

The protocol can further provide notification mechanisms that allowaccessory 112 (or other accessories) to selectively notify controller102 in the event of a state change. Multiple mechanisms can beimplemented, and controller 102 can register, or subscribe, for the mostappropriate notification mechanism for a given purpose. Examples ofnotification mechanisms are described in above-referenced U.S.application Ser. No. 14/614,914.

In some embodiments, communication with a given accessory can be limitedto authorized controllers. The protocol can specify one or moremechanisms (including mechanisms referred to herein as “pair setup” and“pair add”) for establishing a “pairing” between controller 102 and agiven accessory (e.g., door lock accessory 104) under circumstances thatprovide a high degree of confidence that the user intends for controller102 to be able to control accessory 104. Pair setup can include anout-of-band information exchange (e.g., the user can enter a numericalor alphanumeric PIN or passcode provided by accessory 104 into aninterface provided by controller 102) to establish a shared secret. Thisshared secret can be used to support secure exchange of “long-term”public keys between controller 102 and accessory 104, and each devicecan store the long-term public key received from the other, so that anestablished pairing can be persistent. After a pairing is established,controller 102 is considered authorized, and thereafter, controller 102and accessory 104 can go in and out of communication as desired withoutlosing the established pairing. When controller 102 attempts tocommunicate with or control accessory 104, a “pair verify” process canfirst be performed to verify that an established pairing exists (aswould be the case, e.g., where controller 102 previously completed pairsetup with accessory 104). The pair verify process can include eachdevice demonstrating that it is in possession of a long-term private keycorresponding to the long-term public key that was exchanged during pairsetup and can further include establishing a new shared secret orsession key to encrypt all communications during a “pair-verified”session, (also referred to herein as a verified session). During apair-verified session, a controller that has appropriate privileges canperform a “pair add” process to establish another pairing with theaccessory on behalf of another controller. Either device can end apair-verified session at any time simply by destroying or invalidatingits copy of the session key.

In some embodiments, multiple controllers can establish a pairing withthe same accessory (e.g., by performing pair setup or by having apairing added by a controller that previously performed pair setup), andthe accessory can accept and respond to communications from any of itspaired controllers while rejecting or ignoring communications fromunpaired controllers. Examples of pair setup, pair add and pair verifyprocesses, as well as other examples of security-related operations, aredescribed in above-referenced U.S. application Ser. No. 14/614,914.

In some embodiments, controllers (or their users) can be assignedvarious permissions or privileges in regard to the accessories. Forexample, an administrator (or “admin”) privilege may be a highest levelof privilege, and a controller with admin privileges may establishpairings with accessories and control any controllable characteristic ofthe accessory state. In some embodiments, admin privilege may be grantedto the first controller to perform pair setup with a particularaccessory, and after the admin controller performs pair setup, theaccessory can decline to perform pair setup with any other controllers;instead, the admin controller can grant access to other controllers (orother users) by performing pair add. In some embodiments, the admincontroller can specify privileges for each added controller (includingadmin privileges).

It will be appreciated that home environment 100 is illustrative andthat variations and modifications are possible. Embodiments describedherein can be implemented in any environment where a user wishes tocontrol one or more accessory devices using a controller device,including but not limited to homes, cars or other vehicles, officebuildings, campuses having multiple buildings (e.g., a university orcorporate campus), etc. Any type of accessory device can be controlled,including but not limited to door locks, door openers, lighting fixturesor lighting systems, switches, power outlets, cameras, environmentalcontrol systems (e.g., thermostats and HVAC systems), kitchen appliances(e.g., refrigerator, microwave, stove, dishwasher), other householdappliances (e.g., clothes washer, clothes dryer, vacuum cleaner),entertainment systems (e.g., TV, stereo system), windows, window shades,security systems (e.g., alarms), sensor systems, and so on. A singlecontroller can establish pairings with any number of accessories and canselectively communicate with different accessories at different times.Similarly, a single accessory can be controlled by multiple controllerswith which it has established pairings. Any function of an accessory canbe controlled by modeling the function as a service having one or morecharacteristics and allowing a controller to interact with (e.g., read,modify, receive notifications of updates to) the service and/or itscharacteristics. Accordingly, protocols and communication processes usedin embodiments of the technology described herein can be uniformlyapplied in any context with one or more controllers and one or moreaccessories, regardless of accessory function or controller form factoror specific interfaces.

FIG. 2 shows an example network configuration 200. Configuration 200allows controllers 202 to communicate with accessories 204 located inlocal environment 206 (e.g., a home environment) via a coordinator 210.Each controller 202 can be an electronic device owned and/or operated bya user who frequents environment 206 (e.g., a resident of the home or aregular visitor to the home). For example, controller 202 can beresident device (e.g., a desktop computer, tablet computer, streamingmedia device, etc.) that typically stays within (e.g., resides in) localenvironment 206. Controllers 202 can each be similar to controller 102of FIG. 1, and accessories 204 can be similar to various accessoriesshown in FIG. 1.

Accessories 204 can each communicate with a coordinator device (or“coordinator”) 210 that can be located with local environment 206. Asused herein, a “coordinator” can be an electronic device that is capableof operating as a controller of accessories 204 as well as relayingmessages from other controllers (e.g., controllers 202) to accessories204. In some embodiments, coordinator 210 can be an “intelligent” devicethat can coordinate operations among multiple controllers and/oraccessories and is not limited to passively relaying messages.Coordinator 210 can include any device that is capable of presentingitself as a controller to accessories 204 and that is capable ofcommunicating securely with controllers 202. In some embodiments,coordinator 210 can present itself to accessories 204 as a controllerand to controllers 202 as an accessory that provides services forcommunicating with other accessories (e.g., accessories 204); examplesare described in U.S. application Ser. No. 14/725,891. In someembodiments, coordinator 210 can be a device that is expected to stay inlocal environment 206 and that is expected to be powered on andavailable for communication most or all the time. (It is to beunderstood that coordinator 210 can occasionally be unavailable, e.g.,in connection with software or firmware upgrades, power outages, orother intermittent occurrences.) For example, coordinator 210 can beimplemented in a desktop computer, a Wi-Fi or access-point unit, adedicated accessory-control base station, a set-top box for a televisionor other appliance (which can implement coordinator functionality inaddition to interacting with the television or other appliance), or anyother electronic device as desired.

In some embodiments, coordinator 210 and accessories 204 can communicateusing a local area network (LAN), such as a Wi-Fi network and/or apoint-to-point communication medium such as Bluetooth LE. It is to beunderstood that other communication protocols can be used. In someembodiments, controllers 202, accessories 204, and coordinator 210 cansupport a uniform accessory protocol as described above that can besupported using both Wi-Fi and Bluetooth LE as transports.

In the example of FIG. 2, controllers 202(1) and 202(4) are currentlylocated in local environment 206 with accessories 204 and coordinator210. For example, controller 202(1) can be on the same LAN asaccessories 204 and coordinator 210. Controllers 202(2) and 202(3) arecurrently located outside local environment 206 but are connected to acommunication network 208 (e.g., the Internet); such controllers aresaid to be “remote” from accessories 204 and coordinator 210. It is tobe understood that controllers 202 can be mobile devices that aresometimes within local environment 206 and sometimes outside localenvironment 206. Accessories 204 need not be mobile and need not beconnected to communication network 208 (although they can be ifdesired). In some embodiments, coordinator 210 can be connected tocommunication network 208 and can facilitate access to accessories 204by remote controllers 202(2) and 202(3).

In the example shown, controllers 202 can communicate with accessories204 via coordinator 210, and coordinator 210 can be said to act as a“proxy” for accessories 204. Coordinator 210 can communicate directlywith accessories 204(1) and 204(2). In the case of accessory 204(3),coordinator 210 can communicate via “bridge” 212. Bridge 212 can operateto relay commands between a controller and an accessory; in someembodiments, bridge 212 and/or coordinator 210 can also translatebetween different communication protocols used by coordinator 210 orcontroller 202 and accessory 204(3). Further, in some embodiments,bridge 212 can be implemented as a “tunnel” that can provide secureend-to-end communication between coordinator 210 and accessory 204(3).Examples of proxies, bridges, and tunnels are described inabove-referenced U.S. application Ser. No. 14/725,891.

In some implementations of network configuration 200, controllers 202can be configured to communicate with accessories 204 via coordinator210 whenever possible. Thus, as shown, controller 202(1), which is inlocal environment 206, communicates with coordinator 210 rather thandirectly with accessories 204, as do remotely located controllers 202(2)and 202(3). Direct communication between any of controllers 202 andaccessories 204 can be limited, e.g., to situations where coordinator210 is not available. In other embodiments, controllers 202 maycommunicate directly with accessories 204 whenever they happen to be inrange of each other (e.g., on the same Wi-Fi network or within Bluetoothrange). For instance, as shown, controller 202(4) can communicatedirectly with accessory 204(2).

In some embodiments, coordinator 210 can be used to coordinate access bymultiple controllers 202 to multiple accessories 204. For example,rather than establishing a pairing between each controller 202 and eachaccessory 204, controllers 202 can each establish a pairing withcoordinator 210, and coordinator 210 can establish a pairing with eachaccessory 204. The same pair setup and/or pair add processes used toestablish a controller-accessory pairing can also be used to establish acontroller-coordinator pairing, with the coordinator acting in the roleof accessory. For purposes of coordinator-accessory pairing, thecoordinator can assume the role of controller. Thus, coordinator 210 canpresent itself as an accessory when communicating with a controller(e.g., any of controllers 202) and as a controller when communicatingwith an accessory (e.g., accessory 204).

Coordinator 210 can facilitate operation of an accessory networkincluding accessories 204. For example, coordinator 210 can maintain anenvironment model for the accessory network and can provide the model(or portions thereof) to various controllers 202; examples of anenvironment model are described below. Controllers 202 can operateaccessories 204 by interacting with coordinator 210.

In some embodiments, coordinator 210 can manage permissions associatedwith the accessory network or environment model to limit access byspecific controllers 202 to some or all accessories 204. In someembodiments, controllers 202 can preferentially route all requests toaccessories 204 through coordinator 210, and in some embodiments,accessories 204 can be configured to communicate directly only withcoordinator 210 and to ignore requests that come directly fromcontrollers 202. This can allow coordinator 210 to enforce permissionsand other restrictions on access to accessories 204.

Centralizing communication with accessories through coordinator 210 cansimplify management of a controller network and/or accessory network(e.g., controllers 202 and accessories 204 in local environment 206).For example, if a new accessory is acquired, the new accessory need onlyestablish a pairing with coordinator 210 in order to allow allcontrollers 202 to have access to the new accessory. Similarly, if a newcontroller 202 is acquired, the new controller 202 need only establish apairing with coordinator 210 to allow the new controller to have accessto all accessories 204. In an environment with multiple controllers(e.g., a family where the members each have multiple devices) andperhaps dozens of accessories, the time saving can be considerable.

It should be noted that in configuration 200, it is possible that one ormore of the controllers (e.g., controller 202(1)) can be permitted tocommunicate with one or more accessories (e.g., accessory 204(1))indirectly (via coordinator 210) but not directly, regardless of whethercontroller 202(1) is in local environment 206. This might occur, forinstance, if controller 202(1) has established a pairing withcoordinator 210 but not directly with accessory 204(1). In someinstances, this can provide enhanced security; for instance, anaccessory that has a pairing established with coordinator 210 can refuseto establish any other pairings. However, there may be cases wheredirect access is desirable, and establishing a direct pairing between acertain accessory, e.g., accessory 204(1) and one or more controllers202 can be permitted. For example, suppose that accessory 204(1) is adoor lock and controller 202(1) is a mobile phone. If a direct pairingbetween accessory 204(1) and controller 202(1) is established, a usercan use controller 202(1) to lock or unlock accessory 204(1) via directcommunication, thereby locking or unlocking the door. This can beuseful, e.g., in the event that coordinator 210 is temporarilyunavailable. In some embodiments, coordinator 210 can be used toindicate to accessory 204(1) which of controllers 202 are authorized fordirect access, and accessory 204(1) can establish pairings withauthorized controllers 202. In some embodiments, accessory 204(1) can beconfigured to accept direct communication from an authorized controller202 only when coordinator 210 is not available. Thus, the general rulecan be that all communications with accessory 204 go through coordinator210, with exceptions made on a per-accessory and per-controller basis.

Coordinator 210 can operate as an intelligent agent for allowingcontrollers to operate accessories, rather than simply relayingmessages. For example, coordinator 210 can establish a pairing with eachof controllers 202 and a pairing with each accessory 204. Whencontroller 202(1), for example, receives a user request to interact witha specific accessory, e.g., accessory 204(1), controller 202(1) canestablish a first pair-verified session with coordinator 210 and provideits instructions for accessory 204 to coordinator 210 via the firstpair-verified session. Coordinator 210 can receive the instructions,establish a second pair-verified session with accessory 204 and sendappropriate control messages to accessory 204 via the secondpair-verified session. In some embodiments, coordinator 210 can be privyto the content of the instructions, and in some embodiments, themessages sent to accessory 204 need not correspond to the instructionsprovided by controller 202(1). For example, while communicating withcontroller 202(1), coordinator 210 may also be in communication withanother controller (e.g., controller 202(2)). Controllers 202(1) and202(2) may each provide instructions for accessory 204 to coordinator210. Coordinator 210 can analyze the received instructions, e.g., todetect and resolve conflicts such as where controller 202(1) instructscoordinator 210 to turn accessory 204 on while controller 202(2)instructs coordinator 210 to turn accessory 204 off. Coordinator 210 canbe programmed with priority rules or other rules for resolving conflicts(e.g., “on” takes priority over “off”; instructions from a controllerwith admin privilege take precedence over instructions from a controllerwithout admin privilege; etc.). Coordinator 210 can apply the priorityrules to resolve any conflicts and can communicate instructions toaccessory 204 based on the resolution. When a response is received fromaccessory 204, coordinator 210 can determine whether to send acorresponding message (or a different message) to controller 202(1)and/or to controller 202(2).

As another example, coordinator 210 can enforce permissions establishedfor various controllers 202 and/or accessories 204. For example, whenone of controllers 202 sends a request, coordinator 210 can applydecision logic to determine whether the controller 202 that sent therequest has appropriate permission; if not, coordinator 210 can rejectthe request. The decision logic can be as simple or complex as desired;for instance, a controller belonging to a child may be limited as towhich hours of the day or for how long it can operate a particularaccessory (e.g., a TV) while a parent's controller can have unlimitedaccess, or a controller associated with a guest (e.g., a babysitter) maybe restricted to operating a certain subset of the accessories. Thus,coordinator 210 is not limited to acting as a passive relay for messagesbetween controllers and accessories but can actively intervene toresolve conflicting instructions, enforce any limitations that may existon the privileges or permissions granted to particular controllers orusers, and so on.

It will be appreciated that network configuration 200 is illustrativeand that variations and modifications are possible. Any number ofcontrollers and any number of accessories can be included in a networkconfiguration. In some embodiments, coordinator 210 can be replaced witha proxy that relays messages between controllers and accessories withoutnecessarily reading the content of the messages. In some embodiments,coordinator 210 can be omitted entirely. Some or all of accessories 204may be accessible only within the local environment. Further, asdescribed below, different controllers 202 may have different levels ofpermission in regard to accessing accessories 204; for instance, remoteaccess via network 208 may be permitted for some controllers 202 but notfor other controllers 202.

As noted above, coordinator 210 can be particularly useful in thecontext of an automated environment with a number of accessories thatcan be controlled. Examples include homes, cars or other vehicles,office buildings, campuses having multiple buildings, etc. For purposesof illustration, an example of an accessory network implementation for ahome will be described; those skilled in the art with access to thepresent disclosure will understand that similar accessory networks canbe implemented in other automated environments.

In one example of an accessory network, each accessory is connected toone or more controllers, and accessories can be controlled by sendingmessages, e.g., as described in above-referenced U.S. application Ser.No. 14/725,912 and U.S. application Ser. No. 14/614,914. This can beperfectly serviceable for small networks with just a few accessories.However, in some instances, particularly as the number of accessoriesincreases, it can be helpful to establish meaningful (to a user) groupsof accessories that can be managed in a coordinated fashion.Accordingly, certain embodiments of the present technologies describedherein incorporate environment models usable to coordinate controlacross multiple accessories in an accessory network.

As used herein, an environment model can provide various logicalgroupings of the accessories in an environment. For example, a homeenvironment can be modeled by defining “rooms” that can represent roomsin the home (e.g., kitchen, living room, master bedroom, etc.). In somecases, a room in the model need not correspond to a room in the home;for instance, there can be a “front yard” room or an “anywhere” room(which can be used to refer to accessories that are present in the homebut whose location within the home is subject to change or has not beendefined as a room). Each accessory in the home can be assigned to a roomin the environment model, e.g., based on the actual physical location ofthe accessory. Rooms can be grouped into zones based on physical and/orlogical similarities. For instance, an environment model for a two-levelhouse might have an “upstairs” zone and a “downstairs” zone. As anotherexample, an environment model might have a “bedrooms” zone that includesall bedrooms regardless of where they are located. The model can be assimple or complex as desired, e.g., depending on the size and complexityof the environment.

Where an environment model is defined, accessories represented in theenvironment model can be controlled individually or at the level ofrooms, zones, or the whole model. For instance, a user can instruct acontroller or coordinator to turn on all the outside lights or to turnoff all accessories in a specific room.

Other groupings of accessories can also be defined. For example, in someembodiments, a user can augment an environment model by grouping variousaccessories into “service groups” that can include any set ofaccessories the user may desire to control together, at least some ofthe time. A service group can include accessories in any combination ofrooms or zones, and the accessories in a service group can behomogeneous (e.g., all upstairs lights) or heterogeneous (e.g., a light,a fan, and a TV). In some embodiments, a user can provide a singleinstruction to a controller to set the state of an entire service group(e.g., turn the group on or off). While not required, the use of servicegroups can provide another degree of flexibility in coordinating controlover multiple accessories.

In some embodiments, the environment model for a given environment canbe represented as a data object (or set of data objects). Theenvironment model can be created on a controller associated with theenvironment (e.g., a controller with admin privileges) and can be sharedwith other controllers through a synchronization operation. Forinstance, controllers 202 of FIG. 2 can synchronize with a “master” copyof the environment model maintained by coordinator 210 (which canreceive updates from controllers 202), or cloud-based synchronization(in which the master copy is stored in a location accessible via network208 and automatically synchronized with the controllers andcoordinator(s) associated with the environment) can be used.Accordingly, all controllers and coordinators associated with a givenenvironment can have shared access to the same environment model.

Additional examples related to defining and using an environment modelare described in above-referenced U.S. application Ser. No. 14/725,912.It is to be understood that an environment model is not required to makeuse of at least some of the features described below.

FIG. 3 is a block diagram of an example system 300 for managingaccessories. In some implementations, system 300 can include user device302. User device 302 can, for example, correspond to one of controllers202 (e.g., controller 202(1), controller 202(2), etc.), as describedabove with reference to FIG. 2. User device 302 can correspond tocoordinator 210 (e.g., coordinator 116), as described above withreference to FIG. 2. For example, user device 302 can be a computingdevice, such as a laptop computer, tablet computer, smartphone, orwearable device (e.g., a smartwatch, smart glasses, smart clothing,etc.). User device 302 can be a computing device, such as a desktopcomputer, streaming media device, home media server, router, or othercomputing device.

In some implementations, user device 302 can include home application304. For example, home application 304 can be a standalone userapplication or a system application (e.g., tightly integrated with orpart of the operating system) of user device 302. Home application 304can provide graphical user interfaces and other user accessible featuresthat allow the user to interact with, manage, and/or control networkedaccessory devices within a home environment, as described further hereinbelow.

In some implementations, home application 304 can be configured tomanage and control accessories and accessory states. For example, when auser installs or configures an accessory (e.g., accessory 310, accessory320) in the user's home, the smart accessory can broadcast a message(e.g., a Bluetooth signal) advertising the existence of the smartaccessory. Home application 304 can receive the broadcast message andadd the smart accessory to the accessories managed by home application304. For example, home application 304 can receive state informationfrom individual accessories (e.g., accessory 310, accessory 320, etc.)through network 330 (e.g., a WAN, LAN, WLAN, peer-to-peer Wi-Fi,Bluetooth, etc.) and present the state information to the user on adisplay of user device 302. Home application 304 can send commands(e.g., automatically and/or in response to user input) to change thecurrent state of the individual accessories through network 330. Thus,home application 304 can turn on and off smart lights, lock and unlocksmart locks, turn on and off cameras, receive alarms from smokedetectors, and manage other smart accessories and appliances throughoutthe user's home.

In some implementations, home application 304 can manage groups ofaccessories. For example, when managing a home environment, homeapplication 304 can group accessories (e.g., accessory 310 and accessory320, etc.) according to the rooms in the house where the accessories arelocated, as described above. Thus, a user can interact with homeapplication 304 to control all of the accessories in a room as a group.For example, a single user input to home application 304 can cause homeapplication 304 to send a command to each accessory (e.g., accessory310, accessory 320, etc.) in an accessory group (e.g., service group)through network 330 to change the current state (e.g., turn on, turnoff) of all of the accessories assigned to a room.

In some implementations, home application 304 can group accessoriesbased on function, classification, or category. For example, accessoriesrelated to external security (e.g. external lights, door locks, etc.)can be grouped together even though the accessories are not located inthe same room. In some implementations, these service groups can begenerated by home application 304 in response to user input assigningaccessories to specific groups (e.g., to rooms, to functionalcategories, etc.). For example, the user can apply labels (e.g., roomnames, categories, etc.) to accessories and home application 304 canassign the accessories to service groups based on a set of rules forprocessing the labels assigned to the accessories. In someimplementations, home application 304 can automatically groupaccessories according to various criteria, as described further below.In some implementations, home application 304 can group accessoriesbased on a user-defined grouping. In some implementations, homeapplication 304 can group accessories based on related uses. Forexample, home application 304 can learn, based on historical accessorystate change data, which accessories the user typically uses togetherand/or what settings or states the user specifies for the accessoriesand generate service groups, scenes, and/or triggers and/or determineanomalous accessory states based on the learned user behavior, asdescribed in detail below.

In some implementations, user device 302 can include home daemon 305.For example, home daemon 305 can be a daemon or background processrunning on user device 302 that monitors the state of variousaccessories and/or coordinates communication between accessories andother user devices (e.g., other home applications), as described aboveand below. In some implementations, home daemon 305 can be configured tocollect state information, configuration information, and/or featureinformation from various smart accessories and store the accessoryinformation in the appropriate databases (e.g., accessory database 306,accessory state database 308, etc.). When home application 304 requiresaccessory information (e.g., state information, configurationinformation, feature information, accessory control information, etc.)for accessories managed by home application 304, home application 304can request the accessory information from home daemon 305 and homedaemon 305 can obtain the information from the appropriate databases(e.g., accessory database 306, accessory state database 308, etc.) asdescribed below. In some implementations, home daemon 305 can includesome or all of the features of home application 304. For example, homedaemon 305 can learn, based on historical accessory state change data,which accessories the user typically uses together and/or what settingsor states the user specifies for the accessories and generate servicegroups, scenes, and/or triggers and/or determine anomalous accessorystates based on the learned user behavior, as described in detail below.

While many of the features disclosed herein are described as features ofhome application 304, the same or similar features can be performed byhome daemon 305. For example, analysis of historical data, generation ofnew scenes, service groups, and other machine learning activities can beperformed by home daemon 305. Home daemon 305 can then deliver theresults of such analysis (e.g., new scenes, service groups, etc.) tohome application 304 (e.g., on the same device, on a different device,etc.) for presentation to the user.

In some implementations, when user device 302 is configured as acontroller (e.g., controller 202(1), controller 202(2)), user device 302can include home application 304, home daemon 305, accessory database306, and/or accessory state database 306. When user device 302 isconfigured as a coordinator (e.g., coordinator 116, coordinator 210),user device 302 may include a reduced feature set and include homedaemon 305, accessory database 306 and/or accessory state database 308.As described above, as a controller, user device 302 can act as bothcontroller and coordinator using home application 304 and home daemon305.

In some implementations, system 300 can include accessory 310. Forexample, accessory 310 can correspond to one of accessories 204 (e.g.,accessory 204(1)) of FIG. 2. As described above, accessory 310 caninclude logic (e.g., software) and hardware (e.g., integrated circuits,radio frequency transmitters, memory, etc.) that cause accessory 310 todetermine its current state and report its current state to user device302 through network 330. Accessory 310 can include logic and hardwarethat cause accessory 310 to receive commands from user device 302through network 330 that cause accessory 310 to change its current state(e.g., turn on/off, adjust volume, change speed, etc.). For example,accessory 310 can include lights, locks, doorbells, appliances, smokedetectors, carbon monoxide detectors, motion detectors, blinds, garagedoor openers, and/or other electrical devices, appliances, etc., thatmight be in a home, workplace, or other environment.

In some implementations, system 300 can include accessory 320. Forexample, accessory 310 can correspond to one of accessories 204 (e.g.,accessory 204(2)) of FIG. 2. For example, accessory 320 can include thesame or similar features as accessory 310. Accessory 320 can, forexample, be the same type of device as accessory 310. Accessory 320 canbe a different type of device (e.g., a fan vs. a light) and havedifferent features (e.g., fan speed vs. light color) than accessory 310.However, both accessory 310 and accessory 320 can be smart accessoriesthat can communicate with and be managed by home application 304.

In some implementations, user device 302 can include accessory database306. For example, accessory database 306 can include accessoryconfiguration information for accessories (e.g., accessory 310,accessory 320) managed by user device 302. Home application 304 and/orhome daemon 305 can, for example, obtain accessory configurationinformation (e.g., features, APIs, controls, commands, etc.) fromaccessory 310 when home application 304 and/or home daemon 305 connectsto accessory 310 through network 330. For example, accessory 310 cansend its configuration information to home application 304 uponestablishing a connection to home application 304 and/or home daemon 305through network 330. Accessory 310 can send its configurationinformation to home application 304 and/or home daemon 305 in responseto a request for configuration information from home application 304and/or home daemon 305.

In some implementations, system 300 can include resident device 340. Forexample, resident device 340 can have the same or similar features asuser device 302. For example, resident device 340 can include homedaemon 345 corresponding to home daemon 305, accessory database 346corresponding to accessory database 306, and/or accessory state database348 corresponding to accessory state database 308. For example, residentdevice 340 can be a user device (e.g., desktop computer, laptopcomputer, streaming media device, set top box, etc.) that is typicallykept within a home environment. Because resident device 340 is typicallykept within the home environment, resident device 340 can stay connectedto accessories within the home environment when other user devices(e.g., user device 302) cannot. For example, user device 302 may enterand exit the home environment and, accordingly, connect to anddisconnect from accessories and other user devices (e.g., residentdevice 340) within the home environment. Since resident device 340 isconsistently (if not constantly) connected to the accessories and/oruser devices within the home environment, resident device 340 may have amore complete set of historical accessory state data, context data,and/or user device state data than a user device (e.g., user device 302)that may enter and exit the home environment more frequently. Thus,because resident device 340 can consistently collect state data fromvarious accessories and/or user devices, resident device 340 may bebetter suited to analyzing historical accessory state data andgenerating scene suggestions, scene trigger suggestions, and detectinganomalous accessory states than user device 302. After generatingsuggestions or detecting anomalous accessory state, resident device 340can send the suggested scene information, suggested trigger information,and/or anomalous accessory state information to home application 304 onuser device 302 for presentation to the user, as described below.

Monitoring Accessory States

FIG. 4 is an illustration of an example home environment 400 havingvarious smart accessories 402-432. While the description of thetechnologies described herein are described with reference to a home orresidence, a person of ordinary skill in the art will understand thatthe features, processes, algorithms, and mechanisms implemented by thesetechnologies can be easily applied to other contexts such as an office,a warehouse, a garage, or other environment.

In some implementations, home environment 400 can be configured withsmart accessories 402-432. For example, smart accessories 402-432 cancorrespond to accessories 310 and/or 320 of FIG. 3. Smart accessories402-432 can be managed and/or controlled by home application 304 and/orhome daemon 305 on user device 302, as described herein. Similarly,smart accessories 402-432 can be managed and/or controlled by homedaemon 345 on resident device 340, as described herein. For simplicity,the descriptions that follow may describe embodiments, implementations,and/or features with reference to user device 302 and/or homeapplication 304. However, resident device 340 (e.g., home daemon 345)may perform the same or similar embodiments, implementations, and/orfeatures as described herein with reference to home application 304and/or home daemon 305 on user device 302.

In an example scenario (e.g., scenario ‘A’), at the front entrance (e.g.front door) of home environment 400, the owner (i.e., the user of userdevice 302) of home environment 400 has installed an external light 402,an external camera 404, and an external doorbell 406. When a visitorrings doorbell 406, doorbell 406 can send a status message to homeapplication 304 on user device 302 indicating that someone manipulated(e.g., pressed a button) doorbell 406 to cause doorbell 406 to ring. Inresponse to receiving the message, home application 304 can present anotification on the display of user device 302 notifying the user thatdoorbell 406 has been rung. The user can then provide input to homeapplication 304 to turn on external light 402 and camera 404 so that theuser can view the person at the door using a video feed from camera 404presented on the display of user device 302. The user may provide inputto home application 304 to cause home application 304 to unlock the doorusing door lock 403 when the user knows the visitor and wants thevisitor to enter home environment 400.

In another example scenario (e.g., scenario ‘B’), the living room ofhome environment 400 can include lamp 408 and lamp 410. For example,lamp 408 (e.g., a light bulb, light fixture, lamp, etc.) can be anaccessory (e.g. accessory 310) that has various features. Lamp 408 may,for example, simply turn on and off like a normal light. Lamp 408 may beable to illuminate different colors. Lamp 408 may be dimmable such thatlamp 408 can illuminate at different brightness levels. Lamp 410, forexample, can have similar or different features than lamp 408. Forexample, lamp 408 may only be able to turn on and off, while lamp 410might have a dimmer and color selection features. When the user entersthe living room to watch television (e.g., smart television 416 and/orstreaming media device 414), read a book, or play a game, the user canturn on (e.g., when watching television) or off (e.g., when reading orplaying a game) lamps 408 and 410. The user can turn on and off lamps408 and lamp 410 remotely using home application 304 or manually byinteracting with each lamp individually.

As another example scenario (e.g., scenario ‘C’), the living room ofhome environment 400 can include air conditioner controller 412 (e.g., asmart thermostat), streaming media device 414, smart television 416,and/or smart fan 418. When the user watches television in the livingroom, the user may turn on smart television 416, streaming media device414, fan 418, and turn on the home air conditioner using controller 412to make the room nice and cool for watching television. The user canturn on these accessories manually using switches on the accessoriesand/or typical remote controls. The user can turn on these accessoriesusing home application 304 on user device 302. When the user is finishedwatching television, the user can turn off these accessories manuallyusing switches on the accessories and/or typical remote controls. Theuser can turn off these accessories using home application 304 on userdevice 302.

As another example scenario (e.g., scenario ‘D’), in a bedroom of homeenvironment 400 the user may have installed smart lamps 432 and 434 nextto the user's bed. The user's morning routine might be that the userturns on lamp 432 and/or lamp 434 and goes to the kitchen and turns onsmart coffee maker 420 before going to the bathroom and turning on smartlight 422 and smart fan 424 before taking a shower. The user can turn oneach of these accessories manually and individually by interactingphysically with each device. The user can turn on each of theseaccessories using home application 304 on user device 302.

When the user interacts, manipulates, or changes the state of theaccessories (e.g., as described in the scenarios above), each accessorycan report a state change event that identifies its new state (e.g., nowcurrent state) to home application 304 and/or home daemon 305 on userdevice 302. Home application 304 and/or home daemon 305 can store thestate change event information (e.g., accessory state information)received from the accessories in accessory state database 308. Forexample, accessory state database 308 can store for each state changeevent an accessory identifier, a timestamp indicating when the eventoccurred, and/or the new state for the accessory. Thus, accessory statedatabase 308 can store a history of accessory state changes over time.

In some implementations, accessories (e.g., accessory 310) can reportother state information to home application 304 and/or home daemon 305.For example, accessory 310 can send error state information to homeapplication 304 and/or home daemon 305. For example, accessory 310 candetermine a problem with the power supply (e.g., battery level is low,external power disconnected, etc.) for accessory 310 and report thepower supply problem to home application 304 and/or home daemon 305.Accessory 310 can determine a problem with the configuration ofaccessory 310 (e.g., the firmware or software is out of date) and reportthe configuration problem to home application 304 and/or home daemon305. Accessory 310 can determine a security problem (e.g., anunauthorized user attempted to access the accessory) and report thesecurity problem to home application 304 and/or home daemon 305. Asdescribed above, when home application 304 and/or home daemon 305receives information describing a state change event, home application304 can store the state change event data in accessory state database308.

In some implementations, home application 304 and/or home daemon 305 cancollect context information associated with accessory state changes. Forexample, the context information can include a timestamp indicating whenthe accessory state changes occur. The context information can includeinformation describing a duration (e.g., start time, end time, duration,etc.) of a corresponding accessory state. The context information caninclude information describing which user devices are in homeenvironment 400 when the accessory state changes occur. The contextinformation can include detected events that occur within homeenvironment 400. For example, an event can be an accessory state changeevent, as described above. An event can be a user device entering orexiting home environment 400. Each event can include a timestampindicating when the event occurred, for example. Home application 304and/or home daemon 305 can store the collected context information inaccessory state database 308.

Automatically Determining Related Accessories

In some implementations, home application 304 can use the historicalaccessory state information to determine patterns of use and determinerelationships between accessories based on the historical patterns ofuse. For example, home application 304 and/or home daemon 345 mayperiodically analyze the data in accessory state database 308/348 todetermine related accessories and make suggestions for new accessorygroups, scenes, and/or scene triggers, as described further below. Forexample, in scenario ‘A’ described above, the user uses doorbell 406,external light 402, camera 404, and/or lock 403 within a short period oftime of each other to determine who rang the doorbell and/or to let thevisitor in home environment 400. Because these accessories are reportingtheir states and/or state changes to home application 304 and/or homedaemon 305, accessory state database 308 can include state changeentries for light 402, lock 403, camera 404, and/or doorbell 406indicating that all of these accessories change state within a shortperiod of time. For example, the accessory state change entries caninclude an identifier for the accessory, a timestamp for when the statechange occurred, a description of or value corresponding to the statechange (e.g., on, off, locked, unlocked, etc.) and/or the context of thestate change. For example, state change context can include contextinformation such as the time when the state change occurred, a deviceidentifier for the device that caused the state change, deviceidentifiers for devices that were in the home environment (e.g. the sameenvironment as the accessory that changed state), timestamps indicatingwhen devices entered and/or exited the home environment, and/or anyother context information that can be obtained from the devices (e.g.,user device 302, resident device 340, etc.) within the home environment,as may be described herein.

In some implementations, home application 304 (and/or home daemon 305)can determine related accessories based on historical patterns of use.For example, home application 304 can compare the timestamps for entriesin database 308 to determine that the each of these accessories changestate at about the same time (e.g., within a threshold period of time).Based on the timing of the accessory state change entries in database308, home application 308 can determine a historical pattern of use thatindicates that light 402, lock 403, camera 404, and/or doorbell 406 aretypically used together and are, therefore, related.

In some implementations, home application 304 can implement arelatedness algorithm to determine associations between accessoriesbased on recorded (e.g., historical) accessory state changes. Forexample, home application 304 can implement an a priori algorithm (orother shopping cart algorithm) to determine associations betweenaccessories based on accessory state changes over time. For example,accessory state changes that occur within a window of time (e.g., 1minute, 2 minutes, 3 minutes, or other short period of time, etc.) canbe grouped together and considered a single state change transaction.

Home application 304 (or home daemon 305) can analyze the state changetransactions using the relatedness algorithm to determine associationsbetween accessories over time. For example, to determine relatedness,home application 304 can determine two scores: a support score and aconfidence score. The support score can, for example, be used to filterout infrequent transactions and/or patterns that are not significant.The events can be accessory state change events. The events can betrigger events, as described further below. The confidence score canindicate a level of confidence that events are actually related.

To generate the support score, home application 304 can determine thetotal number of events (T) in accessory state database 308 during aperiod of time (e.g., one week, two weeks, one month, etc.). Exampleevents are turning on a light, turning on a coffee maker, etc. Homeapplication 304 can then determine the number of times (N) a group ofevents (e.g., two or more) occur within a small window of time. Forexample, home application 304 can determine the number of times thecoffee maker is turned on within one minute of the light being turned onbased on the accessory state change event data in database 308. Homeapplication 304 can then calculate the ratio N/T to generate the supportscore. For example, if there are 100 total events within database 308during a one week time period and the light and coffee maker are turnedon within one minute of each other 30 times for 60 events total (e.g., 2accessories×30 events each), then the support score is 60/100 (e.g.,0.60 or 60%). If the support score for a group of events is below asupport threshold value (e.g., 0.01, 0.05, etc.), then home application304 can eliminate the group of events from further analysis whendetermining accessory groups, scene suggestions, and/or triggersuggestions. The support score can be used by home application 304 toeliminate infrequent transactions and/or patterns that are notsignificant.

To generate the confidence score, home application 304 can determine thenumber of times (M) a particular accessory event (e.g., light turns on)occurs within database 308 within the period of time described above.For example, home application 304 can determine that the light is turnedon 70 times. Home application 304 can then determine the number of times(N) two events occur within a small window of time. For example, homeapplication 304 can determine the number of times the coffee maker isturned on within one minute of the light being turned on based on theaccessory state change event data in database 308. Home application 304can then calculate the ratio N/M to generate the confidence score. Forexample, if there are 70 events within database 308 during a one weektime period where the light is turned on and 60 events where the lightand coffee maker are turned on within one minute of each, then theconfidence score is 60/70 (0.86 or 86%). If the confidence score isabove a confidence threshold value, then home application can determinea high confidence that the events are related. For example, if homeapplication 304 determines that there is a high confidence that two ormore accessory state change events are related, then the accessories maybe grouped into a service group, into a scene, or one accessory statechange can be determined to trigger another state change event, asdescribed herein.

Similarly, in scenario ‘B’ described above, the user uses lamp 408 andlamp 410 within a short period of time of each other to set up theliving room to watch television or read a book. Because theseaccessories are reporting their states and/or state changes to homeapplication 304 and/or home daemon 305, accessory state database 308 caninclude state change entries for lamp 408 and lamp 410 indicating thatthese accessories change state within a short period of time of eachother. For example, the accessory state change entries can include anidentifier for the accessory, a timestamp for when the state changeoccurred, and a description of the state change (e.g., on, off, dimmerlevel, light color, etc.). Home application 304 can compare thetimestamps for entries in database 308 to determine that each of theseaccessories changes states at about the same time (e.g., within athreshold period of time). Based on the timing of the accessory statechange entries in database 308, home application 304 can determine ahistorical pattern of use that indicates that lamp 408 and lamp 410 aretypically used together and are, therefore, related.

In scenario ‘C’ described above, the user uses air conditionercontroller 412, streaming media device 414, smart television 416, and/orsmart fan 418 within a short period of time of each other to set up theliving room to watch television. Because these accessories are reportingtheir states and/or state changes to home application 304 and/or homedaemon 305, accessory state database 308 can include state changeentries for air conditioner controller 412, streaming media device 414,smart television 416, and smart fan 418 indicating that theseaccessories change state within a short period of time of each other.For example, the accessory state change entries for each accessory caninclude an identifier for the accessory, a timestamp for when the statechange occurred, and a description of the state change (e.g., on, off,television channel, fan speed, streaming media source, etc.). Homeapplication 304 can compare the timestamps for entries in database 308to determine that each of these accessories changes states at about thesame time (e.g., within a threshold period of time). Based on the timingof the accessory state change entries in database 308, home application308 can determine a historical pattern of use that indicates that airconditioner controller 412, streaming media device 414, smart television416, and smart fan 418 are typically used together and are, therefore,related.

In scenario 'D described above, the user uses smart lamp 432, smart lamp434, coffee maker 420, smart light 422 and smart fan 424 within a shortperiod of time of each other as part of the user's morning routine.Because these accessories are reporting their states and/or statechanges to home application 304 and/or home daemon 305, accessory statedatabase 308 can include state change entries for smart lamp 432, smartlamp 434, coffee maker 420, smart light 422 and smart fan 424 indicatingthat these accessories change state within a short period of time ofeach other. For example, the accessory state change entries for eachaccessory can include an identifier for the accessory, a timestamp forwhen the state change occurred, and a description of the state change(e.g., on, off, coffee maker setting, etc.). Home application 304 cancompare the timestamps for entries in database 308 to determine thateach of these accessories changes states at about the same time (e.g.,within a threshold period of time). Based on the timing of the accessorystate change entries in database 308, home application 308 can determinea historical pattern of use that indicates that smart lamp 432, smartlamp 434, coffee maker 420, smart light 422 and smart fan 424 aretypically used together and are, therefore, related.

Generating Service Groups Suggestions

In some implementations, home application 304 and/or home daemon 345 canautomatically generate service group suggestions based on relatedaccessories. For example, when home application 304 determines thataccessories are related, as described above, home application 304 canautomatically create a service group that includes the relatedaccessories. For example, home application 304 can create a servicegroup (e.g., “Front Door Security”) that includes light 402, lock 403,camera 404, and/or doorbell 406, as described in scenario ‘A’ above.Home application 304 can automatically create a service group (e.g.,“Living Room Lights”) that includes lamp 408 and lamp 410, as describedin scenario ‘B’ above. Home application 304 can automatically create aservice group (e.g., “Living Room Entertainment”) that includes airconditioner controller 412, streaming media device 414, smart television416, and smart fan 418, as described in scenario ‘C’ above. Homeapplication 304 can automatically create a service group (e.g., “WakeUp”) that includes smart lamp 432, smart lamp 434, coffee maker 420,smart light 422 and smart fan 424, as described in scenario ‘D’ above.After generating the service group, home application 304 can present aservice group suggestion on a graphical user interface of homeapplication 304. For example, the user can select the service groupsuggestion to approve the service group suggestion and have the servicegroup presented on GUI 500 of FIG. 5.

In some implementations, home application 304 can manage accessories ina service group as a collective (e.g., as a single entity). For example,even though a service group may include multiple accessories, homeapplication can provide a group control that allows the user to adjustthe settings of multiple accessories in a service group with a singleinput. Similarly, notifications generated by an accessory in a servicegroup can be treated and/or presented as a service group notificationrather than an individual accessory notification. By combiningaccessories into a service group, the user can more easily find,control, and interact with accessories to adjust or control the user'senvironment.

Generating Scenes Suggestions

In some implementations, home application 304 and/or home daemon 345 canautomatically generate scenes suggestions based on related accessories.For example, while a service group provides group controls for the userto specify settings for the service group, a scene is configured withsettings so that when a user invokes or selects a scene home application304 automatically adjusts the state of the accessories in the scene tomatch the configured settings for the scene. For example, when creatinga scene, home application 304 can determine that accessories are relatedbased not only on timing, as described above, but also based on thestates of the related accessories. For example, home application 304 candetermine a pattern of use where accessories are used at about the sametime and the settings are the same in each instance of the pattern ofuse over time.

In some implementations, the relatedness algorithm described above canbe used to determine relatedness of accessory states when making scenesuggestions. For example, instead of just analyzing transactions foraccessory identifiers, home application 304 can use the a priorialgorithm to determine the relatedness of accessory states or accessorystate change events by analyzing accessory identifier-state pairs. Anaccessory identifier-state pair can, for example, be a pairing of anidentifier for an accessory and the state of the accessory recorded at aparticular time. For example, home application 304 can determine thatseveral accessory identifier-state pairs are related when the accessoryidentifier-state pairs frequently appear in transactions together. Theaccessory identifier-state pairs for scenario ‘A’ described above canbe, for example, ‘402:on’ (e.g., light 402 is in the ‘on’ state),‘403:unlocked’ (e.g., lock 403 is in the ‘unlocked’ state), ‘404:on’(e.g., camera 404 is in the ‘on’ state), and ‘406:on’ (e.g., doorbell406 is in the ‘on’ state—the doorbell has been rung).

Similarly to the relatedness algorithm described above, home application304 can determine that several accessory identifier-state pairs arerelated when the accessory identifier-state pairs appear together morethan the threshold percentage of time. For example, if the analysis ofthe accessory identifier-state pairs for light 402, lock 403, camera 404and doorbell 406 results in a confidence score that is greater than ascene confidence threshold value (e.g., 90%, 80%, etc.), then homeapplication 304 can determine that the respective states of light 402,lock 403, camera 404 and doorbell 406 are related enough to be includedin a scene suggestion.

However, in addition to the confidence score, home application 304 cananalyze accessory state change data to determine temporal or contextualconsistency between related accessory state changes when generating ascene suggestion. For example, home application 304 can determinewhether accessory state change data in database 308 has a pattern ofaccessory state changes that indicates that a group of accessoriesconsistently changes state at a particular time of day, particular dayof the week, etc. Home application 304 can determine whether accessorystate change data in database 308 has a pattern of accessory statechanges that indicates that a group of accessories consistently changesstate when a particular context occurs (e.g., the user arrives in thehome environment). Thus, if an accessory or group of accessories usuallychange to a particular state at a particular time of day or in responseto a particular context, home application 304 can generate a scenesuggestion for the group of accessories.

For example, in scenario ‘A’ above, when doorbell 406 rings (state=on),the user turns on both light 402 (state=on) and camera 404 (state=on).Home application 304 can create scene (e.g., scene ‘A’) that includeslight 402 and camera 404 because database 308 will have historical statedata that indicates a historical pattern of use that includes turning onlight 402 and camera 404 at about the same time. A short time later, theuser may unlock lock 403 to let a visitor in the user's home. Thus, thestate of lock 403 can be ‘unlocked’. Home application 304 can configurescene ‘A’ so that when the user invokes or selects scene ‘A’, homeapplication 304 will cause light 402 and camera 404 to turn on. Homeapplication 304 may exclude lock 403 from scene ‘A’ because lock 403 isa security-related accessory that might require further user input inorder to change state. After generating and configuring a scene, homeapplication 304 can present a scene suggestion on a graphical userinterface of home application 304. For example, the user can select thescene suggestion to approve the scene suggestion and have the scenepresented on GUI 500 of FIG. 5, as described further below.

As another example, in scenario ‘C’ above, the user may set airconditioner controller 412 to 60 degrees, set smart fan 418 to mediumspeed, turn on streaming media device 414, and turn on smart television416 and set the television to the input for streaming media device 414.Home application 304 can create scene (e.g., scene ‘C’) that includesair conditioner controller 412, smart fan 418, streaming media device414, and smart television 416 because database 308 will have historicalstate data that indicates a historical pattern of use that includeschanging the setting of air conditioner controller 412 to 60 degrees,changing the setting of smart fan 418 to medium speed, turning onstreaming media device 414, and turning on smart television 416 andsetting the input to the input corresponding to streaming media device414 at about the same time. Home application 304 can configure scene ‘C’so that when the user invokes or selects scene ‘C’, home application 304will set air conditioner controller 412 to 60 degrees, set smart fan 418to medium speed, turn on streaming media device 414, and turn on smarttelevision 416 and set the television to the input for streaming mediadevice 414.

Thus, home application 304 can automatically generate scenes (and scenesuggestions) that recreate a previous state of related accessories thatare typically used or adjusted at about the same time. Stateddifferently, home application 304 can automatically generate a scenebased on a historical pattern of use that is based not only on whichaccessories are used within a short period of time (e.g., 1 minute, 3minutes, 7 minutes, etc.) of each other but also based on the settingsof each accessory when the accessories are used together.

Generating Trigger Suggestions

In some implementations, home application 304 can automatically generatesuggestions for triggers. For example, a trigger can be an event thatwhen detected by home application 304 causes home application 304 toautomatically invoke a corresponding scene. A trigger can be an eventthat when detected by home application 304 causes home application 304to automatically change the state of an accessory. A trigger can bedetermine based on historical accessory state change data and/or contextdata that indicates an event that usually precedes or follows a sceneinvocation, state changes related to an accessory group, or a statechange related to an individual accessory. For example, the historicaldata may indicate that a particular accessory state change always ormostly follows a particular event such that the event is a reliablepredictor of the particular accessory state change. For example, theuser (or user device) entering the home environment can be a goodpredictor that the air conditioner will soon be turned on. Thehistorical data may indicate that a particular accessory state changealways or mostly precedes a particular event. For example, the user maytypically lock the front door before going to sleep. User device 302 canpredict when the user will go to sleep and automatically lock the frontdoor in anticipation of the user sleeping.

In some implementations, the trigger event can be an accessory statechange event. For example, the accessory state change event can be thatthe garage door opens, the front doorbell is rung, the thermostatdetects a specified temperature, or any other accessory state change.The trigger event can correspond to a change in location of a userdevice. For example, user device 302 can report device stateinformation, including the device's current location, to homeapplication 304 and/or home daemon 305. Home application 304 candetermine a trigger for a scene, accessory group, or individualaccessory based on the location of user device 302. For example, userdevice 302 can send location information indicating that user device 302has left the user's work location. In response to detecting that userdevice 302 has left the user's work location, home application 304 cantrigger a ‘welcome home’ scene that turns on the air conditioner at theuser's home and turns on the radio to the user's favorite program orstation. The trigger event can be a time-based event. For example, theevent can be that the current time is 6 pm. When home application 304detects that the current time is 6 pm (or other time), home application304 can invoke the corresponding scene, or change the state of anaccessory group or individual accessory.

In some implementations, the trigger event can correspond to a change inthe home environment. For example, home daemon 345 on resident device340 can detect when different user devices (e.g., user device 302)connect to and/or disconnect from the home environment and/or residentdevice 340. For example, a family of three may live within a homemanaged by home application 304 and/or home daemon 345. As each memberof the family enters and leaves the home environment with theirrespective devices, the devices can connect and disconnect with theaccessories and other user devices (e.g., resident device 340) withinthe home environment. Thus, resident device 340 can detect when devicesenter and leave the home environment and/or determine which user devicesare within the home environment at any time. This context information(e.g., which devices are within the home environment) and/or deviceconnecting and disconnecting events can be used as triggers for scenes.For example, when home daemon 345 detects that someone (e.g., some userdevice) has left the home environment and nobody else (e.g., no otheruser-specific device) is left within the home environment, then homedaemon 345 can use the leaving home event as a trigger for a scene thatturns off lights, air conditioner, coffee maker, and locks all doors andwindows. The context information can be used to trigger state changes inaccessories associated with accessory groups (e.g., service groups)and/or individual accessories.

In some implementations, home application 304 can use contextinformation to generate conditional triggers. For example, homeapplication 304 can determine that unlocking the front door lock is atrigger for turning on an entrance light and adjusting the thermostat.However, turning on the entrance light can be conditioned on contextinformation that indicates the door lock was unlocked after sunset. Forexample, home application 304 can determine that the light is typicallyonly turned on when it is dark outside and/or inside the house (e.g., asindicated by light sensor data or sunset/sunrise times). Further, homeapplication 304 can determine based on context information (e.g.,temperature sensor data reported by a thermostat) that the thermostat istypically only adjusted when the temperature inside the house is above76 degrees or below 66 degrees Fahrenheit. Thus, the context datacollected by home application 304 can be used to generate triggers thatare conditioned upon specific context, as determined based on historicalcontext data and/or historical accessory state change event data.

In some implementations, the trigger event can correspond to othercontext information reported by user device 302. For example, userdevice 302 may be configured to determine a current user activity basedon sensor data generated by user device 302. For example, user device302 may be configured to determine when the user is sleeping, awake,running, walking, sitting, driving, etc., based on motion sensor data,sound sensor data, light sensor data, and/or other sensor data collectedby user device 302. User device 302 can send this user activity data tohome application 304 and/or home daemon 305. User device 302 can sendthis user activity data to home daemon 345 on resident device 340. Thus,home application 304 and/or home daemon 305/345 can determine triggersfor scenes, accessory groups, and/or individual accessories based on thereported user activity data. While triggers are described herein withreference to scene triggers, triggers for service groups and individualaccessories can be generated similarly to scene triggers.

In some implementations, the relatedness algorithm described above canbe used to determine relatedness of trigger events to scenes, servicegroups, and/or individual accessories when making trigger suggestions.For example, instead of just analyzing transactions for accessoryidentifiers and/or accessory states to determine scene suggestions, asdescribed above, home application 304 can use the relatedness algorithmto determine the relatedness of detected events (e.g., detectedcontexts) to scenes (e.g., accessories and/or accessory states). Forexample, accessory state database 308 can include context data and/orevent data that is timestamped to indicate a time at which the contextwas reported and/or when the reported event occurred. These contextsand/or events can be included with the accessory state change eventswhen grouping the accessory state change events into transactions.

Home application 304 can determine that detected events and/or contextsare related to a scene when the detected events frequently appear intransactions with a suggested or determined scene, as described above.For example, home application 304 and/or home daemon 345 can determine atime based trigger when the user typically (e.g., above a thresholdpercentage of the time) invokes a particular scene around 7 pm. Homeapplication 304 and/or home daemon 345 can determine a time basedtrigger relative to some event. For example, home application 304 candetermine that the user invokes a particular scene or adjusts aparticular accessory (e.g., a light on the front porch) 30 minutes aftersunset. Home application 304 and/or home daemon 345 can determine alocation based trigger when the user typically (e.g., above a thresholdpercentage of the time) invokes a particular scene when the user arrivesat the user's home (e.g., home environment). Home application 304 and/orhome daemon 345 can determine a user activity based trigger when theuser typically (e.g., above a threshold percentage of the time) invokesa particular scene after the user wakes, one hour before the usersleeps, 30 minutes after the user exercises, or after the user performssome other detectable activity. For example, home application 304 candetermine that a detected event or context is a trigger for a scene whenthe confidence score for the candidate trigger event and scene is abovea trigger confidence threshold value (e.g., 80%, 95%, etc.), asdescribed above.

After home application 304 and/or home daemon 345 determines a suggestedtrigger, home application 304 can present the suggested scene trigger ona graphical user interface of home application 304, as described below.For example, if home daemon 345 on resident device 340 generates thesuggested scene trigger (or suggested scene, or suggested service group,as described above), home daemon 345 can send the suggested scenetrigger to home application 304 on user device 302 for presentation tothe user.

FIG. 5 is an example graphical user interface 500 presented by homeapplication 304. For example, graphical user interface (GUI) 500 can bethe initial graphical interface (e.g., home screen, default GUI, etc.)presented when home application 304 is invoked on user device 302. GUI500 can be presented on a display of user device 302, for example. GUI500 can be an interface for managing accessories within a home, office,or other building. Home application 304 can be configured to manageaccessories within different homes, offices, buildings, environments,etc., and the user can provide input to home application 304 to switchbetween environments to view the accessories, service groups, scenes,etc., configured for each environment. The descriptions that follow willdescribe various implementations, features, processes, and graphicaluser interfaces of home application 304 with respect to the user's home(e.g., labeled “My Home”), however the implementations described hereincan be applied to any environment, not just the user's home.

In some implementations, GUI 500 can include status element 501. Forexample, status element 501 can present accessory status data reportedby the various accessories (e.g., accessory 310, accessory 320)connected to (e.g., registered with, paired with, etc.) home application304, as described above. Status element 501 can present accessory statusdata as determined by home application 304. For example, homeapplication 304 may determine that accessory 320 is unreachable orunresponsive when the connection between home application 304 andaccessory 320 is broken (e.g., the network is unavailable, the accessoryhas lost power, etc.). Status element 501 can provide the user a quickview of the current status of the user's home accessories. Statuselement 501 is described in further detail below in reference to FIG.21A and FIG. 21B.

In some implementations, GUI 500 can include graphical elements 502-508representing accessories that have been paired with home application304. For example, if home application 304 has been configured tocommunicate with or has previously established a connection with anaccessory, the accessory can be represented on GUI 500. However, becauseof the limited display space available on most mobile devices, GUI 500may only display accessories that the user has designated as favorites.If an accessory is currently connected to home application 304, thecorresponding graphical element (e.g., graphical element 502) canpresent a full, bright color representation of the accessory. If anaccessory is not current connected to home application 304, thecorresponding graphical element can present a subdued, greyed, orotherwise diminished representation of the accessory.

In some implementations, graphical elements 502-508 can appear as tilesthat include information (e.g., accessory type, location, status, etc.)describing the corresponding accessory. For example, graphical element502 can represent and describe a television (e.g., smart television 416)that is located in the user's family room. Graphical element 504 canrepresent and describe a ceiling fan that is located in the user'sbedroom. Graphical element 506 can represent and describe an overheadlight that is located in the user's kitchen. Graphical element 508 canrepresent and describe a radio that is located in the user's bedroom.The user can select (e.g., tap) any of the tiles to turn on and off(e.g., toggle back and forth) the corresponding accessory. The user canselect (e.g., touch and hold, apply pressure, etc.) any of the tiles tocause home application 304 to present a detailed view (not shown) of thecorresponding accessory, including presenting any additional controlsthe user can select to manipulate the settings of the correspondingaccessory. In some implementations, GUI 500 can include graphicalelement 510 for adding an accessory, creating a new service group,and/or creating a new scene.

In some implementations, GUI 500 can include graphical element 512 forselecting a home environment to present on GUI 500. For example, a usermay have different home environments configured in home application 304.The user may have a primary home environment, a vacation homeenvironment, an office environment, etc. Each one of these homeenvironments can be configured with smart accessories and managed byhome application 304, as described herein. As depicted in FIG. 5, homeapplication 304 is currently presenting accessories and statusinformation for the user's primary home environment labeled “My Home.”However, the user can select graphical element 512 to cause homeapplication 304 to present or represent a different home environment onGUI 500.

In some implementations, GUI 500 can include graphical element 514 forpresenting scene suggestions. For example, home application 304 and/orhome daemon 305 on user device 302 can be configured to generate scenesuggestions and/or scene trigger suggestions, as described above. Homedaemon 345 on resident device 340 can be configured to generate scenesuggestions and/or scene trigger suggestions. When home daemon 305 onuser device 302 and/or home daemon 345 on resident device 340 generatescene suggestions and/or scene trigger suggestions, home daemon 305and/or home daemon 345 can send the suggestions to home application 304on user device 302. Home application 304 can present the scenesuggestions and/or scene trigger suggestions on a graphical userinterface of home application 304, as described further below.

FIG. 6 illustrates an example graphical user interface 600 forpresenting suggested service groups, scenes, and/or scene triggers. Forexample, GUI 600 can be presented by home application 304 on a displayof user device 302 in response to receiving a user selection ofgraphical element 514 of FIG. 5.

In some implementations, GUI 600 can include service group suggestions.For example, GUI 600 can include graphical element 602 representing asuggested service group. As described above, home application 304 and/orhome daemon 305/345 can generate a service group suggestions bydetermining which accessories are related based on timing of accessorystate changes. After home application 304 generates the service groupsuggestion or receives the service group suggestion from home daemon305/345, home application 304 can present graphical element 602representing the service group suggestion on GUI 600. For example,graphical element 602 can represent a front door security service groupthat includes light 402, lock 403, camera 404, and/or doorbell 406.

Similarly, GUI 600 can include scene suggestions. For example, GUI 600can include graphical elements 604-610 representing suggested scenes. Asdescribed above, home application 304 and/or home daemon 305/345 cangenerate scene suggestions by determining which accessories are relatedbased on timing of accessory state changes and/or the accessory statevalues. After home application 304 generates the scene suggestions orreceives the scene suggestions from home daemon 305/345, homeapplication 304 can present graphical elements 604-610 representing therespective scene suggestions on GUI 600. For example, GUI 600 caninclude a “movie night” scene suggestions that includes air conditionercontroller 412 (e.g., a smart thermostat), streaming media device 414,smart television 416, and/or smart fan 418 and is configured to changethe state of these accessories to set up the home environment to watchmovies on television 416.

In some implementations, GUI 600 can include suggested triggers forscenes. As described above, home application 304 and/or home daemon305/345 can generate scene trigger suggestions by determining whichaccessories are related based on timing of detected events (e.g. contextchanges), timing of accessory state changes and/or the accessory statevalues. After home application 304 generates the scene triggersuggestions or receives the scene trigger suggestions from home daemon305/345, home application 304 can present graphical elements 612-614representing the respective scene suggestions on GUI 600. For example,GUI 600 can include a “Goodnight” scene trigger suggestion that includesa “Goodnight” scene (e.g., specifying state values for light 402, lock403, lamp 408, lamp 410, etc.) and a “Goodnight” scene trigger (e.g., aparticular time, a particular event, etc.) that will cause homeapplication 304 to automatically invoke the “Goodnight” scene. Asillustrated by FIG. 6, GUI 600 can include both a scene suggestion 608for the “Goodnight” scene and a scene trigger for the “Goodnight” scene.For example, the user may select to approve the “Goodnight” scene butthe user may not wish to have the “Goodnight” scene automaticallyinvoked by the scene trigger 612.

In some implementations, home application 304 can receive user input toaccept or reject a suggested service group, scene, or scene trigger. Forexample, each graphical element 602-614 representing a suggested servicegroup, scene, or scene trigger can include graphical elements (e.g.,graphical elements 620, 622) that a user can select to accept or rejecta suggested item. For example, the user can select graphical element 620to reject the corresponding suggested service group. When homeapplication 304 receives a selection of graphical element 620, homeapplication 304 can delete (e.g., remove, hide, etc.) graphical element602 from GUI 600 and store data that will prevent home application 304from presenting the corresponding service group again in the future. Insome implementations, the user can select graphical element 622 toaccept the suggested service group. For example, when home application304 receives user input accepting a suggested service group, scene, orscene trigger, the corresponding service group, scene, or scene triggercan be removed from GUI 600 and presented on GUI 500 of FIG. 5.

In some implementations, home application 304 and/or home daemon 345 canadjust the thresholds (e.g., service group confidence threshold, sceneconfidence threshold, trigger confidence threshold, etc.) for therelatedness algorithm based on whether a user accepts or rejects asuggested item. For example, each type of suggested item (e.g., servicegroup, scene, scene trigger, etc.) can have a different relatednessthreshold. For example, the service group confidence threshold (e.g.,60%) can be lower than the scene confidence threshold (e.g., 70%). Thescene confidence threshold can be lower than the trigger confidencethreshold (e.g., 80%). When a user accepts or rejects a suggestedservice group, scene, or scene trigger, home application 304 can adjustthe confidence threshold value for the type of item rejected oraccepted. For example, when home application 304 receives user inputrejecting a suggested trigger, home application 304 can adjust thetrigger confidence threshold upward from 80% to 81%. Thus, fewer scenetriggers will be suggested in the future. When home application 304receives user input accepting a scene, home application 304 can adjustthe scene confidence threshold down from 70% to 69% so that homeapplication 304 will suggest more scenes.

In some implementations, home application 304 can present details for asuggested scene or scene trigger. For example, a user can select one ofgraphical elements 604-610 to cause home application 304 to present aGUI (e.g., GUI 700) for viewing and editing details about thecorresponding suggested scene. The user can select one of graphicalelements 612-614 to cause home application 304 to present a GUI (e.g.,GUI 800) for viewing and editing details about the corresponding scenesuggestion.

FIG. 7 illustrates an example graphical user interface 700 forpresenting details about a scene. For example, GUI 700 can be presentedby home application 304 on a display of user device 302 in response tothe user selecting graphical element 604 of FIG. 6.

In some implementations, GUI 700 can include graphical element 702(e.g., a text input box, a text control, etc.) for specifying a name forthe selected scene. For example, the user can provide input to a virtualkeyboard (not shown) to edit the name of the suggested scene. Forexample, the suggested scene may be automatically named by homeapplication 304 and/or home daemon 345 initially. The user can provideinput to graphical element 702 to rename the scene.

In some implementations, GUI 700 can present graphical elements (e.g.,704-714) representing accessories managed by home application 304.Accessories that are already part of the scene (e.g., accessories704-710) can be marked with a graphical element (e.g., marked with thecheck mark) so that the user can easily see which accessories arealready in the scene. The user can edit the suggested scene by selectingaccessories managed by home application 304 and presented on GUI 700.For example, the user can select (e.g., tap) the accessories that theuser wishes to add to the suggested scene or remove from the suggestedscene. GUI 700 can receive the user input selecting graphical elements704-714 and add or remove the corresponding accessories to the suggestedscene. For example, when the user selects the suggested scene (e.g.,element 604 on GUI 600), the suggested scene can include lamp 408, lamp410, smart television 416, smart fan 418, and two Bluetooth speakersfrom the user's living room. The user can select graphical elements704-714 to remove the corresponding accessories from the suggestedscene. The user can select graphical elements 716 and/or 718 to add thecorresponding accessories to the suggested scene.

In some implementations, home application 304 can automaticallydetermine the initial settings for each accessory included in a scene.For example, accessories in the suggested scene can be configured withaccessory states based on the accessory state history data, as describedabove. As accessories are added to a scene, home application 304 candetermine the current state of the accessory. For example, if lamp 408is added to the “Movie Night” scene in response to the user selectinggraphical element 704, home application 304 can determine the currentstate (e.g., on or off, illumination level, illumination color, etc.) oflamp 408. Home application 304 can save the current state of lamp 408 asthe scene setting (e.g., scene state) for lamp 408 in the “Movie Night”scene.

Similarly, when smart television 416 is added to the “Movie Night” scenein response to the user selecting graphical element 710, homeapplication 304 can determine the current state (e.g., on or off,channel, input source, volume, etc.) of smart television 416. Homeapplication 304 can save the current state of smart television 416 asthe scene setting for smart television 416 in the “Movie Night” scene.If the user wishes to adjust the initial settings for a specificaccessory, the user can select (e.g., long touch, pressure touch, etc.)the graphical element (e.g., graphical element 704) corresponding to thespecific accessory to invoke a graphical user interface (not shown) foradjusting the settings of the specific accessory.

Later, when the “Movie Night” scene is invoked by the user, homeapplication 304 can send messages to each accessory associated with the“Movie Night” scene specifying the scene setting saved for the accessoryto cause each accessory to assume the saved scene setting. For example,if the “Movie Night” scene setting for lamp 408 is 50% illumination andblue color, home application 304 can send a message to lamp 408indicating 50% illumination and blue color. In response to receiving themessage from home application 304, lamp 408 can generate a blue light at50% illumination. Similarly, if the “Movie Night” scene setting forsmart television 416 is 70% volume and channel 4, home application 304can send a message to smart television 416 indicating 70% volume andchannel 4. In response to receiving the message from home application304, smart television 416 can adjust its volume to 70% and change itschannel to channel 4. When the user is done reviewing and/or editing thesuggested scene, the user can select graphical element 720 to cause homeapplication 304 to present GUI 600 of FIG. 6.

FIG. 8 illustrates an example graphical user interface 800 forpresenting details about a scene trigger. For example, GUI 800 can bepresented by home application 304 in response to receiving a userselection of graphical element 614 of GUI 600. GUI 800 can presentsimilar information and graphical elements as GUI 700. However, GUI 800can also present trigger event information and/or selectable options sothat the user can view and/or edit the events that will automaticallyinvoke the corresponding scene.

In some implementations, GUI 800 can include graphical element 802(e.g., a text input box, a text control, etc.) for specifying a name forthe selected scene trigger. For example, the user can provide input to avirtual keyboard (not shown) to edit the name of the suggested scenetrigger. For example, the suggested scene trigger may be automaticallynamed by home application 304 and/or home daemon 345 initially. The usercan provide input to graphical element 802 to rename the scene trigger.

In some implementations, GUI 800 can include scene trigger options. Forexample, GUI 800 can include scene trigger options 810-816. For example,scene trigger options scene trigger options 810-816 can be automaticallydetermined by home application 304 and/or home daemon 345 based onhistorical accessory state change data and/or historical event data(e.g., context data), as described above. Home application 304 can, forexample, automatically determine multiple potential trigger events basedon the relatedness algorithms described above. Home application 304 canselect the most probable trigger event (e.g., the trigger event thatoccurs in the historical data the most number of times) as the triggerfor the suggested scene trigger. For example, if the historicalaccessory state data and historical event data show that the user ismore likely to invoke the “Wake Up!” scene (or adjust the correspondingaccessories) at 9 am than in response to other events (e.g., when screenon user's device is lit, when the user wakes, when the user's alarmsounds, etc.), then home application 304 will automatically selectoption 810 as the scene trigger, as indicated by FIG. 8.

In some implementations, home application 304 can configure a scene withmultiple triggers. For example, while home application 304 can suggest atime-based scene trigger (e.g., 9 am, option 810) based on historicalaccessory state change information and context information, homeapplication 304 can present other (e.g., additional) scene triggers thatmeet the trigger confidence threshold described above. Thus, if thereare four different triggers that meet or exceed the confidence thresholdfor triggers, home application 304 can present all four scene triggeroptions 810-816 on GUI 800.

When home application 304 presents multiple trigger options, the usercan select multiple triggers to cause home application 304 toautomatically invoke the corresponding scene. For example, thetime-based trigger option 810 can be automatically selected by homeapplication 304. The user activity-based trigger option 812 (e.g., theuser wakes from sleeping) can be selected by the user instead of or inaddition to the automatically selected trigger option 810. The user mayselect an accessory state-based trigger option 816 that causes homeapplication 304 to invoke the corresponding scene when the specifiedaccessory state change (e.g., an alarm triggered) is detected. Whenmultiple triggers are selected, home application 304 can invoke thecorresponding scene the first time a selected trigger is detected withina period of time (e.g., 24 hours). When multiple triggers are selected,home application 304 can invoke the corresponding scene every time aselected trigger is detected.

In some implementations, GUI 800 can include graphical elements 830-834representing accessories managed by home application 304. For example,graphical elements 830-834 can represent accessories selected for thepresented scene and/or scene trigger. When the scene is invoked, homeapplication 304 can automatically adjust the state of the accessoriescorresponding to graphical elements 830-834, as described above.

FIG. 9 illustrates an example graphical user interface 900 forpresenting user approved scene suggestions and triggers. For example,GUI 900 can correspond to GUI 500 of FIG. 5. However, in FIG. 9, theuser has accepted or approved of some scene suggestions that are nowpresented on GUI 900.

In some implementations, FIG. 9 includes graphical elements 902-906representing scene suggestions and/or scene trigger suggestions that theuser has accepted. For example, graphical element 902 can correspond toa “Movie Night” scene. Graphical element 904 can correspond to a“Goodnight” scene trigger. For example, both graphical elements 902 and904 can correspond to scenes. The user can select graphical element 902and/or graphical element 904 to invoke the corresponding scene. However,because graphical element 904 includes a scene trigger, graphicalelement 904 can present information describing the trigger (e.g. “at 10pm”) for the scene that will cause home application 304 to automaticallyinvoke the corresponding scene “Goodnight” at 10 pm.

FIG. 10 illustrates an example graphical user interface 1000 forpresenting anomalous accessory state data. For example, GUI 100 can bepresented by home application 304 in response to home application 304and/or home daemon 345 detecting an anomalous accessory state. Forexample, an anomalous accessory state can be detected when a currentaccessory state (e.g., given a current context, other accessory states,etc.) does not correspond to an expected accessory state. Homeapplication 304 can determine the expected accessory state similarly todetermining scene suggestions and/or scene trigger suggestions, asdescribed above. For example, home application 304 can analyzehistorical accessory state change data and/or context data using therelatedness algorithm (e.g., a priori) algorithm described above todetermine related accessories, accessory states, and/or contexts. Homeapplication 304 can detect patterns in the historical accessory statedata and/or context data that represent normal states, activities,behaviors, etc., with respect to accessories and contexts. Homeapplication 304 can detect when the current activity, accessory states,and/or contexts do not comport with normal activities, states, and/orcontexts and present a notification to the user to draw the user'sattention to the anomalous accessory state.

For example, using the relatedness algorithm, home application 304 candetermine that when the garage door is opened (e.g., change in accessorystate), it is usually opened at 6 pm (e.g., context) and stays open for5 minutes (e.g., context derived from open and close events). When homeapplication 304 determines that the garage door has been opened for 10minutes, home application 304 can present a notification 1002 indicatingthat the garage door has been open for longer than the expected amountof time. In other words, the expected state of the garage door after 5minutes of being open is the ‘closed’ state and since the garage door isstill open, the garage door is in an anomalous state based on thehistorical data.

As another example, home application 304 and/or home daemon 345 candetermine based on historical accessory state data and/or context data,that the oven and/or stove is never turned on when there are no people(e.g., based on the presence or absence of their respective devices) inthe home environment. As described above, home daemon 345 can determinethat the people who live in the home environment have left the homeenvironment, detect that the stove and/or oven are turned on, anddetermine, based on historical data, that the ‘on’ state of the stove oroven is not a normal state when there are no people in the homeenvironment. When home daemon 345 determines that the current state(e.g., ‘on’) of the oven or stove is not the expected state of the ovenor stove for the current context (e.g., everyone is out of the house),home daemon 345 can send a notification to home application 304 to causehome application 304 to present notification 1002 on user device 302.

As yet another example, home application 304 can determine anomalousaccessory states based on a missing event. For example, home applicationcan determine that the front door lock is usually unlocked around 3 pmwhen children arrive home from school. If home application 304determines or detects that the front door lock is locked around 3 pm,home application 304 can monitor the locked state of the front door fora period of time (e.g., 15 minutes, 20 minutes, etc.) and generate ananomaly notification if the front door lock remains locked. Thus, theparents of the children can be notified of a situation that may preventtheir children from entering the home after school.

In some implementations, home daemon 345 can determine anomalousaccessory states based on a probability that the accessory state wouldoccur at a particular time. For example, home daemon 345 can determine,based on historical accessory state data, the probability that anaccessory would have a particular state at a particular time. Forexample, home daemon 345 can divide a 24 hour period into 5 minutetimeslots. For each timeslot, home daemon 345 can determine theprobability (e.g., using well known prediction methods) that anaccessory would have a particular state. For example, home daemon 345can determine the probability that the front door lock would beunlocked. Home daemon 345 can determine the probability that the garagedoor would be open. Home daemon 345 can determine the probability thatthe oven would be on. If home daemon 345 determines that the currentstate of an accessory has a very low probability (e.g., lower than somethreshold value) of occurring in the current timeslot, then home daemon345 can detect an anomalous accessory state and send a notification touser device 302 indicating the anomalous accessory state.

In some implementations, home application 304 and/or home daemon 345 candetermine exceptions to anomaly determinations. For example, home daemon345 can determine a shift in accessory state change patterns thatindicate a time shift in user behavior. For example, home daemon 345 candetermine that the current state of some accessories is not the expectedstate of those accessories at the current time, as described above.However, home daemon 345 can determine that the accessory statescorrespond to a scene that is typically invoked earlier (or later) inthe day. Home daemon 345 can recognize the pattern of current accessorystates and determine that the pattern has merely shifted from thetypically earlier time to a later time and is, therefore, not ananomaly.

In some implementations, home daemon 345 can determine anomalyexceptions based on contextual information received from user device302. For example, home daemon 345 can receive calendar data, maps data,motion data, location data, etc., from user device 302. Home daemon 345can detect an anomaly, such as the garage door being open at a strangetime. Home daemon 345 can compare the time of anomalous accessory stateto calendar data that indicates a late meeting, maps data that shows theuser was just driving home, and/or location data indicating that theuser was away from the house and is now home. Based on this contextualdata, home daemon 345 can determine that the anomalous accessory statecorresponds to the user coming home and is not, in fact, an anomaly.

In some implementations, notification 1002 can describe an anomalydetected by home application 304 and/or home daemon 345. For example,when the user has left the garage open for an unusually long time,notification 1002 can present a message that describes the anomaly.

In some implementations, notification 1002 can include graphical element1004 for returning the accessory to the normal accessory state. Forexample, when notification 1002 indicates that the user has left thegarage open for an unusually long time, notification 1002 can presentgraphical element 1002 that when selected by the user, causes the homeapplication 304 to cause garage door opener (e.g., the accessory) toclose the garage door. Thus, the accessory can be put back into thenormal state for the current time and/or context. If the user does notwish to change the anomalous state of the accessory, the user can selectgraphical element 1006 to hide or dismiss notification 1002.

Example Processes

FIG. 11 is a flow diagram of an example process 1100 for generatingscene suggestions. For example, home application 304 and/or home daemon305 on user device 302 can perform process 1100 to generate scenesuggestions based on historical accessory state change data collected byhome application 304 and/or home daemon 305. In some implementations,home daemon 345 on resident device 340 can perform process 1100 togenerate scene suggestions based on historical accessory state changedata received from accessories and other user devices. To simplify thedescription of process 1100, the process will be described in terms ofsteps being performed by user device 302. However, resident device 340can perform the steps of process 1100 described below and send the scenesuggestions to user device 302 so that user device 302 can present thescene suggestions to the user.

At step 1102, user device 302 can receive accessory event dataindicating accessory state changes. For example, home application 304and/or home daemon 305 can receive accessory state change event datafrom accessories in home environment 400. When accessories change state(e.g., light is turned on/off, lock is locked/unlocked, garage isopened/closed, etc.), the accessories can report their respective statesto home application 304 and/or home daemon 305 running on user device302. Home application 304 and/or home daemon 305 can store the accessorystate change event data in accessory state database 308. For example,the accessory state change event data can include an accessoryidentifier, a user identifier identifying the user who initiated thestate change, a device identifier for the user device that caused thestate change, the new device state, a timestamp indicating when thestate change occurred, and/or a home environment identifier indicatingto which home environment the accessory is connected. Home application304 and/or home daemon 305 can collect accessory state change event dataover time and generate a historical record of accessory state changeevent data in database 308.

At step 1104, user device 302 can identify a pattern of accessory statechanges that indicates a relationship between accessories. For example,home application 304 and/or home daemon 305 can periodically analyze thehistorical accessory state change event data in database 308 todetermine which accessories and/or accessory states are associated orrelated to each other.

In some implementations, the accessories can be related based on time.For example, accessories that frequently change state at about the sametime may be related based on use. Accessories that are adjusted by auser at about the same time may be adjusted to create an environment(e.g., a scene) for some sort of user activity, such as movie watching,sleeping, working, exercising, etc. Home application 304 and/or homedaemon 305 can perform a relatedness algorithm (e.g., modified a priorialgorithm, shopping cart algorithm, etc.) to determine the relatednessof the accessories and/or accessory states, as described above. Therelatedness algorithm can be performed to determine a pattern of use ora pattern of accessory state changes over time that indicates that agroup of accessories and/or accessory states are related and recur overtime.

At step 1106, user device 302 can generate a suggested scene based onthe identified pattern of accessory state changes. For example, a scenecan include accessories and respective accessory states determined bythe relatedness algorithm described above. The scene can be invoked tocause the state of the accessories associated with the scene to changeto the respective states specified by the scene for each accessory. Ascene can be generated based on a group of accessories that is relatedand has been determined by home application 304 and/or home daemon 305to frequently recur, as determined by the historical accessory statechange event data stored in database 308. At this point, since the scenehas not been accepted or approved by the user, the scene is just asuggestion and not yet available for invocation by the user or executionby user device 302.

At step 1108, user device 302 can present the scene suggestion on adisplay of user device 302. For example, home daemon 305/345 can sendthe suggested scene to home application 304. Home application 304 canpresent the suggested scene on a GUI of home application 304, asdescribed above. Home application 304 can receive user input indicatingthat the user wishes to accept or reject the scene suggestion. When theuser accepts the scene suggestion, the scene can be invoked by the userand executed by home application 304. When the user rejects the scenesuggestion, home application 304 can hide the scene and prevent the userfrom invoking the scene in the future.

FIG. 12 is a flow diagram of an example process 1200 for generatingscene trigger suggestions. For example, home application 304 and/or homedaemon 305 on user device 302 can perform process 1200 to generate scenetrigger suggestions based on historical accessory state change dataand/or context data collected by home application 304 and/or home daemon305. In some implementations, home daemon 345 on resident device 340 canperform process 1200 to generate scene trigger suggestions based onhistorical accessory state change data and/or context data received fromaccessories and other user devices. To simplify the description ofprocess 1200, the process will be described in terms of steps beingperformed by resident device 340. However, user device 302 can performsimilar steps as user device 302 described below.

At step 1202, resident device 340 can receive accessory event data froma plurality of accessories. For example, resident device 340 can be auser device (e.g., user device 302) that always, or almost always, stayswithin home environment 400. Since resident device 340 is moreconsistently, if not constantly, connected to the accessories withinhome environment 400, home daemon 345 can collect a more comprehensiveamount of accessory state change event data over time and store theaccessory state change event data in accessory state database 348.

At step 1204, resident device 340 can receive context data from aplurality of user devices. For example, as a coordinator device (e.g.,coordinator 210), resident device 340 can collect context data from userdevices (e.g., user device 302) that enter home environment 400. Userdevice 302 can, for example, send context data that includes locationdata, user activity data, calendar data, map data, motion data, andother types of state data generated or collected by the sensors and/orsoftware of user device 302 to home daemon 345. Home daemon 345 canstore the context data in accessory state database 348. The context datacan include a user identifier identifying the user who is associatedwith the device, a device identifier for the user device, the contextdata, and/or a timestamp indicating when the context data was determinedor detected.

At step 1206, resident device 340 can identify a pattern of accessorystate changes that indicates a relationship between accessories. Forexample, home daemon 345 can periodically analyze the historicalaccessory state change event data in database 348 to determine whichaccessories and/or accessory states are associated or related to eachother.

In some implementations, the accessories can be related based on time.For example, accessories that frequently change state at about the sametime may be related based on use. Accessories that are adjusted by auser at about the same time may be adjusted to create an environment(e.g., a scene) for some sort of user activity, such as movie watching,sleeping, working, exercising, etc. Home daemon 345 can perform arelatedness algorithm (e.g., modified a priori algorithm, shopping cartalgorithm, etc.) to determine the relatedness of the accessories and/oraccessory states, as described above. The relatedness algorithm can beperformed to determine a pattern of use or a pattern of accessory statechanges over time that indicates that a group of accessories and/oraccessory states are related and recur over time.

At step 1208, resident device 340 can generate a scene based on thepattern of accessory state changes. For example, a scene can includeaccessories and respective accessory states determined by therelatedness algorithm described above. The scene can be invoked to causethe state of the accessories associated with the scene to change to therespective states specified by the scene for each accessory. A scene canbe generated based on a group of accessories that is related and hasbeen determined by home daemon 345 to frequently recur, as determined bythe historical accessory state change event data stored in database 308.At this point, since the scene has not been accepted or approved by theuser, the scene is just a suggestion and not yet available forinvocation by the user or execution by user device 302.

At step 1210, resident device 340 can determine a trigger for the scenebased on context data. For example, home daemon 345 can analyze thecontext data in database 348 to determine a time based trigger, locationbased trigger, event based trigger, or some other context trigger. Homedaemon 345 can analyze the context data using the relatedness algorithmdescribed above. For example, in this case, the relatedness algorithm(e.g., the a priori algorithm, shopping cart algorithm, etc.) can takeinto account accessory states and context when determining a trigger forthe scene.

At step 1212, resident device 340 can generate a scene triggersuggestion. For example, home daemon 345 can generate a scene triggerthat includes the determined scene and the determined trigger conditionor context. As described above, the scene can be configured withaccessories and corresponding accessory states such that, when the sceneis invoked, the invoking device (e.g., user device 302) sends commandsto each of the accessories to cause the accessories to transition totheir respective accessory states. The trigger condition can be acontext that precedes or follows the invocation of the scene. The scenetrigger can be used to configure home application 304 on user device 302to automatically invoke the corresponding scene or change the state ofan individual accessory. At this point, since the scene trigger has notbeen accepted or approved by the user, the scene trigger is just asuggestion and not yet available for execution by user device 302.

At step 1214, resident device 340 can cause the scene trigger suggestionto be presented on a display of user device 302. For example, aftergenerating the trigger suggestion, home daemon 345 can send the scenetrigger suggestion to home application 304 on user device 302. When thescene trigger suggestion is received, home application 304 can presentthe scene trigger suggestion on a GUI of home application 304, asdescribed above.

FIG. 13 is a flow diagram of an example process 1300 for determiningaccessory state anomalies. For example, home application 304 and/or homedaemon 305 on user device 302 can perform process 1300 to determineaccessory state anomalies based on historical accessory state changedata and/or context data collected by home application 304 and/or homedaemon 305. In some implementations, home daemon 345 on resident device340 can perform process 1200 to determine accessory state anomaliesbased on historical accessory state change data and/or context datareceived from accessories and other user devices. When home daemon 345determines accessory state anomalies, home daemon 345 can send anotification describing the anomalies to home application 304 on userdevice 302. Home application 304 can then present a graphicalnotification describing the accessory state anomaly on a display of userdevice 302.

At step 1302, user device 302 can receive accessory event dataindicating accessory state changes. For example, home application 304and/or home daemon 305 can receive accessory state change event datafrom accessories in home environment 400. When accessories change state(e.g., light is turned on/off, lock is locked/unlocked, garage isopened/closed, etc.), the accessories can report their respective statesto home application 304 and/or home daemon 305 running on user device302. Home application 304 and/or home daemon 305 can store the accessorystate change event data in accessory state database 308. For example,the accessory state change event data can include an accessoryidentifier, a user identifier identifying the user who initiated thestate change, a device identifier for the user device that caused thestate change, the new device state, a timestamp indicating when thestate change occurred, and/or a home environment identifier indicatingto which home environment the accessory is connected. Home application304 and/or home daemon 305 can collect accessory state change event dataover time and generate a historical record of accessory state changeevent data in database 308.

At step 1304, user device 302 can identify the normal state ofaccessories for a corresponding context. For example, home application304 and/or home daemon 305 can analyze historical accessory state changeevent data stored in accessory state database 308 to determine what thenormal (e.g., usual, typical, most likely, most probable, etc.) state isfor each accessory in a particular context. For example, the context canbe a time of day. The context can be the number of people (e.g., asindicated by user devices) in the home environment. The context can belocation of user devices in or outside of the home environment, or anyother context as may be described herein. For example, home daemon 305can determine that the normal state of the garage door at 2 pm isclosed. Home daemon 305 can determine that the normal state of the frontdoor lock at lam is locked. Home daemon 305 can determine that thenormal state of the oven when nobody is home is off.

At step 1306, user device 302 can determine the current state of anaccessory. As described above at step 1302, home application 304 and/orhome daemon 305 receive accessory state change event data indicating thestates of various accessories as they change state. Since this statechange event data is stored in database 308, home daemon 305 can obtainthe current state of the accessory from database 308.

At step 1308, user device 302 can determine the current operatingcontext for an accessory. For example, home daemon 305 can determine thelocation of user devices with respect to the home environment. Homedaemon 305 can determine the current time. Home daemon 305 can determinethat a particular event just occurred.

At step 1310, user device 302 can determine that the current state isdifferent than the expected state for the current context. For example,home daemon 305 can determine that the current context is lam and thefront door lock is unlocked instead of being locked as is expected atlam. Home daemon 305 can determine that the current context is 2 pm andthe garage door is open instead of being closed as is expected at 2 pm.Home daemon 305 can determine that the current context is that allpersonal user devices are outside of the home environment and the ovenis turned on instead of being turned off as is expected when everyone isoutside of the home environment.

In some implementations, home daemon 305 can determine whether ananomalous accessory state falls into an exception that would indicatethat the anomalous accessory state is not anomalous. For example, homedaemon 305 can determine that a normal accessory state has shifted laterin the day when home daemon 305 determines that the user woke up laterthan normal. Home daemon 305 can determine that the normal accessorystate has shifted when other related accessory states have been shiftedat the same time as the anomalous accessory state. Thus, a group ofanomalous accessory states independently or in combination with a changein user behavior can be used by home daemon 305 to identify an exceptionthat would make an anomalous accessory state normal.

At step 1312, user device 302 can generate a notification indicating thedetermined accessory state anomaly. For example, home daemon 305 cangenerate a notification or a message that describes the anomalousaccessory state. Home daemon 305 can send the notification or message tohome application 304 to cause home application 304 to present agraphical notification that includes a description of the anomalousaccessory state on a display of user device 302.

In some implementations, home application 304 can detect duration basedanomalies using timers. For example, home application 304 can determinethat the normal duration for the front door lock to be unlocked is 5minutes. When a user or user device unlocks the front door, homeapplication 304 can set a timer for the expected duration of theunlocked state (e.g., 5 minutes). Alternatively, home application 304can set a time for the expected duration of the unlocked state plus somepadding for flexibility (e.g., 5 minutes+2 minutes). When the timerexpires, home application 304 can determine the current state front doorlock. If the front door lock is still unlocked, home application 304 canpresent a notification describing the anomalous front door lock state tothe user, as described above.

Graphical User Interfaces

This disclosure above describes various Graphical User Interfaces (GUIs)for implementing various features, processes or workflows. These GUIscan be presented on a variety of electronic devices including but notlimited to laptop computers, desktop computers, computer terminals,television systems, tablet computers, e-book readers, smart phones, andwearable devices (e.g., a smart watch, smart glasses, etc.). One or moreof these electronic devices can include a touch-sensitive surface and/orpressure sensitive surface. The touch-sensitive surface can processmultiple simultaneous points of input, including processing data relatedto the pressure, degree or position of each point of input. Suchprocessing can facilitate gestures with multiple fingers, includingpinching and swiping. Such processing can facilitate distinguishingtypes of inputs based on length of touch and pressure applied so thatdifferent responses, operations, and/or features can be provided basedon the type of touch input. Thus, a tap, long touch, and pressure touchcan all be interpreted as different types of input to illicit differenttypes of responses from the computing devices described herein.

When the disclosure refers to “select” or “selecting” user interfaceelements in a GUI, these terms are understood to include clicking or“hovering” with a mouse or other input device over a user interfaceelement, or touching, tapping or gesturing with one or more fingers orstylus on a user interface element. User interface elements can bevirtual buttons, menus, selectors, switches, sliders, scrubbers, knobs,thumbnails, links, icons, radio buttons, checkboxes and any othermechanism for receiving input from, or providing feedback to a user.

Privacy

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used todeliver targeted content that is of greater interest to the user.Accordingly, use of such personal information data enables calculatedcontrol of the delivered content. Further, other uses for personalinformation data that benefit the user are also contemplated by thepresent disclosure.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information data will comply withwell-established privacy policies and/or privacy practices. Inparticular, such entities should implement and consistently use privacypolicies and practices that are generally recognized as meeting orexceeding industry or governmental requirements for maintaining personalinformation data private and secure. For example, personal informationfrom users should be collected for legitimate and reasonable uses of theentity and not shared or sold outside of those legitimate uses. Further,such collection should occur only after receiving the informed consentof the users. Additionally, such entities would take any needed stepsfor safeguarding and securing access to such personal information dataand ensuring that others with access to the personal information dataadhere to their privacy policies and procedures. Further, such entitiescan subject themselves to evaluation by third parties to certify theiradherence to widely accepted privacy policies and practices.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, in the caseof advertisement delivery services, the present technology can beconfigured to allow users to select to “opt in” or “opt out” ofparticipation in the collection of personal information data duringregistration for services. In another example, users can select not toprovide location information for targeted content delivery services. Inyet another example, users can select to not provide precise locationinformation, but permit the transfer of location zone information.

Example System Architectures

FIG. 14 shows a simplified block diagram of an example systemarchitecture for controller 1400. Controller 1400 can implement any orall of the controller functions, behaviors, and capabilities describedherein, as well as other functions, behaviors, and capabilities notexpressly described. Controller 1400 can include processing subsystem1410, storage device 1412, user interface 1414, communication interface1416, secure storage module 1418, and cryptographic logic module 1420.Controller 1400 can also include other components (not explicitly shown)such as a battery, power controllers, and other components operable toprovide various enhanced capabilities. In various embodiments,controller 1400 can be implemented in a desktop computer, laptopcomputer, tablet computer, smart phone, other mobile phone, wearablecomputing device, or other systems having any desired form factor.Further, as noted above, controller 1400 can be implemented partly in abase station and partly in a mobile unit that communicates with the basestation and provides a user interface.

Storage device 1412 can be implemented, e.g., using disk, flash memory,or any other non-transitory storage medium, or a combination of media,and can include volatile and/or non-volatile media. In some embodiments,storage device 1412 can store one or more application and/or operatingsystem programs to be executed by processing subsystem 1410, includingprograms to implement various operations described above as beingperformed by a controller. For example, storage device 1412 can store auniform controller application that can read an accessory descriptionrecord and generate a graphical user interface for controlling theaccessory based on information therein (e.g., as described inabove-referenced U.S. application Ser. No. 14/614,914). In someembodiments, portions (or all) of the controller functionality describedherein can be implemented in operating system programs rather thanapplications. In some embodiments, storage device 1412 can also storeapps designed for specific accessories or specific categories ofaccessories (e.g., an IP camera app to manage an IP camera accessory ora security app to interact with door lock accessories). Storage device1412 can also store other data produced or used by controller 1400 inthe course of its operations, including trigger data objects and/orother data pertaining to an environment model.

User interface 1414 can include input devices such as a touch pad, touchscreen, scroll wheel, click wheel, dial, button, switch, keypad,microphone, or the like, as well as output devices such as a videoscreen, indicator lights, speakers, headphone jacks, or the like,together with supporting electronics (e.g., digital-to-analog oranalog-to-digital converters, signal processors, or the like). A usercan operate input devices of user interface 1414 to invoke thefunctionality of controller 1400 and can view and/or hear output fromcontroller 1400 via output devices of user interface 1414.

Processing subsystem 1410 can be implemented as one or more integratedcircuits, e.g., one or more single-core or multi-core microprocessors ormicrocontrollers, examples of which are known in the art. In operation,processing system 1410 can control the operation of controller 1400. Invarious embodiments, processing subsystem 1410 can execute a variety ofprograms in response to program code and can maintain multipleconcurrently executing programs or processes. At any given time, some orall of the program code to be executed can be resident in processingsubsystem 1410 and/or in storage media such as storage device 1412.

Through suitable programming, processing subsystem 1410 can providevarious functionality for controller 1400. For example, in someembodiments, processing subsystem 1410 can implement various processes(or portions thereof) described above as being implemented by acontroller. Processing subsystem 1410 can also execute other programs tocontrol other functions of controller 1400, including applicationprograms that may be stored in storage device 1412. In some embodiments,these application programs may interact with an accessory, e.g., bygenerating messages to be sent to the accessory and/or receivingresponses from the accessory. Such interactions can be facilitated by anaccessory management daemon and/or other operating system processes,e.g., as described above.

Communication interface 1416 can provide voice and/or data communicationcapability for controller 1400. In some embodiments communicationinterface 1416 can include radio frequency (RF) transceiver componentsfor accessing wireless voice and/or data networks (e.g., using cellulartelephone technology, data network technology such as 3G, 4G/LTE, Wi-Fi,other IEEE 802.11 family standards, or other mobile communicationtechnologies, or any combination thereof), components for short-rangewireless communication (e.g., using Bluetooth and/or Bluetooth LEstandards, NFC, etc.), and/or other components. In some embodimentscommunication interface 1416 can provide wired network connectivity(e.g., Ethernet) in addition to or instead of a wireless interface.Communication interface 1416 can be implemented using a combination ofhardware (e.g., driver circuits, antennas, modulators/demodulators,encoders/decoders, and other analog and/or digital signal processingcircuits) and software components. In some embodiments, communicationinterface 1416 can support multiple communication channels concurrentlyor at different times, using the same transport or different transports.

Secure storage module 1418 can be an integrated circuit or the like thatcan securely store cryptographic information for controller 1400.Examples of information that can be stored within secure storage module1418 include the controller's long-term public and secret keys 1422(LTPKC, LTSKC as described above), and a list of paired accessories 1424(e.g., a lookup table that maps accessory ID to accessory long-termpublic key LTPKA for accessories that have completed a pair setup orpair add process as described above).

In some embodiments, cryptographic operations can be implemented in acryptographic logic module 1420 that communicates with secure storagemodule 1418. Physically, cryptographic logic module 1420 can beimplemented in the same integrated circuit with secure storage module1418 or a different integrated circuit (e.g., a processor in processingsubsystem 1410) as desired. Cryptographic logic module 1420 can includevarious logic circuits (fixed or programmable as desired) that implementor support cryptographic operations of controller 1400, including any orall cryptographic operations described above. Secure storage module 1418and/or cryptographic logic module 1420 can appear as a “black box” tothe rest of controller 1400. Thus, for instance, communication interface1416 can receive a message in encrypted form that it cannot decrypt andcan simply deliver the message to processing subsystem 1410. Processingsubsystem 1410 may also be unable to decrypt the message, but it canrecognize the message as encrypted and deliver it to cryptographic logicmodule 1420. Cryptographic logic module 1420 can decrypt the message(e.g., using information extracted from secure storage module 1418) anddetermine what information to return to processing subsystem 1410. As aresult, certain information can be available only within secure storagemodule 1418 and cryptographic logic module 1420. If secure storagemodule 1418 and cryptographic logic module 1420 are implemented on asingle integrated circuit that executes code only from an internalsecure repository, this can make extraction of the information extremelydifficult, which can provide a high degree of security. Otherimplementations are also possible.

FIG. 15 shows a simplified block diagram of an example systemarchitecture for accessory 1500. Accessory 1500 can implement any or allof the accessory functions, behaviors, and capabilities describedherein, as well as other functions, behaviors, and capabilities notexpressly described. Accessory 1500 can include storage device 1528,processing subsystem 1530, user interface 1532, accessory-specifichardware 1514, communication interface 1536, secure storage module 1538,and cryptographic logic module 1540. Accessory 1500 can also includeother components (not explicitly shown) such as a battery, powercontrollers, and other components operable to provide various enhancedcapabilities.

Accessory 1500 is representative of a broad class of accessories thatcan be operated by a controller such as controller 1400, and suchaccessories can vary widely in capability, complexity, and form factor.Various accessories may include components not explicitly shown in FIG.15, including but not limited to storage devices (disk, flash memory,etc.) with fixed or removable storage media; video screens, speakers, orports for connecting to external audio/video devices; camera componentssuch as lenses, image sensors, and controls for same (e.g., aperture,zoom, exposure time, frame rate, etc.); microphones for recording audio(either alone or in connection with video recording); and so on.

Storage device 1528 can be implemented, e.g., using disk, flash memory,or any other non-transitory storage medium, or a combination of media,and can include volatile and/or non-volatile media. In some embodiments,storage device 1528 can store one or more programs (e.g., firmware) tobe executed by processing subsystem 1530, including programs toimplement various operations described above as being performed by anaccessory, as well as operations related to particular accessorybehaviors. Storage device 1528 can also store an accessory object oraccessory definition record that can be furnished to controller devices,e.g., during device discovery as described in above-referenced U.S.application Ser. No. 14/614,914. Storage device 1528 can also storeaccessory state information and any other data that may be used duringoperation of accessory 1500.

Processing subsystem 1530 can include, e.g., one or more single-core ormulti-core microprocessors and/or microcontrollers executing programcode to perform various functions associated with accessory 1500. Forexample, processing subsystem 1530 can implement various processes (orportions thereof) described above as being implemented by an accessory,e.g., by executing program code stored in storage device 1528.Processing subsystem 1530 can also execute other programs to controlother functions of accessory 1500. In some instances programs executedby processing subsystem 1530 can interact with a controller (e.g.,controller 1400), e.g., by generating messages to be sent to thecontroller and/or receiving messages from the controller.

User interface 1532 may include user-operable input devices such as atouch pad, touch screen, scroll wheel, click wheel, dial, button,switch, keypad, microphone, or the like, as well as output devices suchas a video screen, indicator lights, speakers, headphone jacks, or thelike, together with supporting electronics (e.g., digital-to-analog oranalog-to-digital converters, signal processors, or the like). Dependingon the implementation of a particular accessory 1500, a user can operateinput devices of user interface 1532 to invoke functionality ofaccessory 1500 and can view and/or hear output from accessory 1500 viaoutput devices of user interface 1532. Some accessories may provide aminimal user interface or no user interface. at all. Where the accessorydoes not have a user interface, a user can still interact with theaccessory using a controller (e.g., controller 1400).

Accessory-specific hardware 1534 can include any other components thatmay be present in accessory 1500 to enable its functionality. Forexample, in various embodiments accessory-specific hardware 1534 caninclude one or more storage devices using fixed or removable storagemedia; GPS receiver; power supply and/or power management circuitry; acamera; a microphone; one or more actuators; control switches;environmental sensors (e.g., temperature sensor, pressure sensor,accelerometer, chemical sensor, etc.); and so on. It is to be understoodthat any type of accessory functionality can be supported by providingappropriate accessory-specific hardware 1534 and that accessory-specifichardware can include mechanical as well as electrical or electroniccomponents.

Communication interface 1536 can provide voice and/or data communicationcapability for accessory 1500. In some embodiments communicationinterface 1536 can include radio frequency (RF) transceiver componentsfor accessing wireless voice and/or data networks (e.g., using cellulartelephone technology, data network technology such as 3G, 4G/LTE, Wi-Fi,other IEEE 802.11 family standards, or other mobile communicationtechnologies, or any combination thereof), components for short-rangewireless communication (e.g., using Bluetooth and/or Bluetooth LEstandards, NFC, etc.), and/or other components. In some embodimentscommunication interface 1536 can provide wired network connectivity(e.g., Ethernet) in addition to or instead of a wireless interface.Communication interface 1536 can be implemented using a combination ofhardware (e.g., driver circuits, antennas, modulators/demodulators,encoders/decoders, and other analog and/or digital signal processingcircuits) and software components. In some embodiments, communicationinterface 1536 can support multiple communication channels concurrentlyor at different times, using the same transport or different transports.

Secure storage module 1538 can be an integrated circuit or the like thatcan securely store cryptographic information for accessory 1500.Examples of information that can be stored within secure storage module1538 include the accessory's long-term public and secret keys 1542(LTPKA, LTSKA as described above), and a list of paired controllers 1544(e.g., a lookup table that maps controller ID to controller long-termpublic key LTPKC for controllers that have completed a pair setup orpair add process as described above). In some embodiments, securestorage module 1538 can be omitted; keys and lists of paired controllerscan be stored in storage device 1528.

In some embodiments, cryptographic operations can be implemented in acryptographic logic module 1540 that communicates with secure storagemodule 1538. Physically, cryptographic logic module 1540 can beimplemented in the same integrated circuit with secure storage module1538 or a different integrated circuit (e.g., a processor in processingsubsystem 1530) as desired. Cryptographic logic module 1540 can includevarious logic circuits (fixed or programmable as desired) that implementor support cryptographic operations of accessory 1500, including any orall cryptographic operations described above. Secure storage module 1538and/or cryptographic logic module 1540 can appear as a “black box” tothe rest of accessory 1500. Thus, for instance, communication interface1536 can receive a message in encrypted form that it cannot decrypt andcan simply deliver the message to processing subsystem 1530. Processingsubsystem 1530 may also be unable to decrypt the message, but it canrecognize the message as encrypted and deliver it to cryptographic logicmodule 1540. Cryptographic logic module 1540 can decrypt the message(e.g., using information extracted from secure storage module 1538) anddetermine what information to return to processing subsystem 1530. As aresult, certain information can be available only within secure storagemodule 1538 and cryptographic logic module 1540. If secure storagemodule 1538 and cryptographic logic module 1540 are implemented on asingle integrated circuit that executes code only from an internalsecure repository, this can make extraction of the information extremelydifficult, which can provide a high degree of security. Otherimplementations are also possible.

Accessory 1500 can be any electronic apparatus that interacts withcontroller 1400. In some embodiments, controller 1400 can provide remotecontrol over operations of accessory 1500 as described above. Forexample controller 1400 can provide a remote user interface foraccessory 1500 that can include both input and output controls (e.g., adisplay screen to display current status information obtained fromaccessory 1500 and an input control such as a touchscreen overlay toallow changes to the status information). Controller 1400 in variousembodiments can control any function of accessory 1500 and can alsoreceive data from accessory 1500.

It will be appreciated that the system configurations and componentsdescribed herein are illustrative and that variations and modificationsare possible. It is to be understood that an implementation ofcontroller 1400 can perform all operations described above as beingperformed by a controller and that an implementation of accessory 1500can perform any or all operations described above as being performed byan accessory. A proxy, bridge, tunnel, or coordinator can combinecomponents of controller 1400 and accessory 1500, using the samehardware or different hardware as desired. The controller and/oraccessory may have other capabilities not specifically described herein(e.g., mobile phone, global positioning system (GPS), broadband datacommunication, Internet connectivity, etc.). Depending onimplementation, the devices can interoperate to provide anyfunctionality supported by either (or both) devices or to providefunctionality that is partly implemented in each device. In someembodiments, a particular accessory can have some functionality that isnot accessible or invocable via a particular controller but isaccessible via another controller or by interacting directly with theaccessory.

Further, while the controller and accessory are described herein withreference to particular blocks, it is to be understood that these blocksare defined for convenience of description and are not intended to implya particular physical arrangement of component parts. Further, theblocks need not correspond to physically distinct components. Blocks canbe configured to perform various operations, e.g., by programming aprocessor or providing appropriate control circuitry, and various blocksmight or might not be reconfigurable depending on how the initialconfiguration is obtained. Embodiments of the present technologiesdescribed herein can be realized in a variety of apparatus includingelectronic devices implemented using any combination of circuitry andsoftware.

Further Embodiments

While the technologies described herein have been described with respectto specific embodiments, one skilled in the art will recognize thatnumerous modifications are possible. Controller networks and/oraccessory networks can include as many or as few devices as desired. Useof a proxy or coordinator is not required; regardless of the number ofaccessories or number of controllers, it is always possible (at least inprinciple) to establish pairings between each controller and eachaccessory and to have all controllers operate by controlling accessoriesdirectly. Where an accessory-network model (e.g., an environment model)is provided, each controller can obtain a copy of the model (e.g., viasynchronization) and can provide access to the model through its userinterface.

Further, where proxies or controllers are present, it can be but neednot be the case that all controllers are permitted to access allaccessories via the proxy or controller. Some controllers might berestricted from accessing accessories when not within the localenvironment, and some accessories might require that controllers accessthem directly rather than through a proxy or coordinator.

Embodiments of the present technologies described herein can be realizedusing any combination of dedicated components and/or programmableprocessors and/or other programmable devices. The various processesdescribed herein can be implemented on the same processor or differentprocessors in any combination. Where components are described as beingconfigured to perform certain operations, such configuration can beaccomplished, e.g., by designing electronic circuits to perform theoperation, by programming programmable electronic circuits (such asmicroprocessors) to perform the operation, or any combination thereof.Further, while the embodiments described above may make reference tospecific hardware and software components, those skilled in the art willappreciate that different combinations of hardware and/or softwarecomponents may also be used and that particular operations described asbeing implemented in hardware might also be implemented in software orvice versa.

Computer programs incorporating various features of the presenttechnologies described herein may be encoded and stored on variouscomputer readable storage media; suitable media include magnetic disk ortape, optical storage media such as compact disk (CD) or DVD (digitalversatile disk), flash memory, and other non-transitory media. (It isunderstood that “storage” of data is distinct from propagation of datausing transitory media such as carrier waves.) Computer readable mediaencoded with the program code may be packaged with a compatibleelectronic device, or the program code may be provided separately fromelectronic devices (e.g., via Internet download or as a separatelypackaged computer-readable storage medium).

Thus, although the technologies described herein have been describedwith respect to specific embodiments, it will be appreciated that thedisclosure is intended to cover all modifications and equivalents withinthe scope of the following claims.

What is claimed is:
 1. A method comprising: receiving, by a residentdevice, accessory event data that indicates accessory state changesoccurring at a plurality of accessory devices; based at least in part onthe event data, identifying, by the resident device, a pattern ofaccessory state changes that indicate an expected state for an accessoryin a corresponding context; automatically generating, by the residentdevice, a plurality of scenes that include at least a subset of theplurality of accessory devices based at least in part on the pattern ofaccessory state changes, wherein the plurality of scenes are invoked,states of the subset of the plurality of accessory devices areautomatically adjusted; determining, by the resident device, a currentstate of the accessory; determining, by the resident device, a currentcontext for the accessory; determining, by the resident device, whetherthe current state of the accessory is different from the expected stateof the accessory for the current context; and in accordance with adetermination that the current state of the accessory is different fromthe expected state of the accessory for the current context: generating,by the resident device, an accessory state anomaly notification thatindicates the current state of the accessory is unexpected for thecurrent context; determining, by the resident device, whether theaccessory and the current state of the accessory are part of a scene ofthe plurality of scenes, wherein the scene has been invoked at a time ofa day; in accordance with a determination that the accessory and thecurrent state of the accessory are not part of the scene: sending, bythe resident device, the accessory state anomaly notification to atleast one user device of a plurality of different user devices; and inaccordance with a determination that the accessory and the current stateof the accessory are part of the scene, wherein the resident devicerecognizes a pattern of the current state of the accessory anddetermines that the pattern is shifted from the time of the day to adifferent time; preventing, by the resident device, the accessory stateanomaly notification from being sent.
 2. The method of claim 1, whereinthe pattern of accessory state changes includes an expected duration ofthe expected state in the corresponding context; and further comprising:determining, by the resident device, a current duration of the currentstate of the accessory; determining, by the resident device, that thecurrent state of the accessory is different from the expected state ofthe accessory for the current context when the current duration of thecurrent accessory state exceeds the expected duration of the expectedstate.
 3. The method of claim 1, wherein receiving, by the residentdevice, the accessory event data includes receiving the accessory eventdata from the at least one user device, where the at least one userdevice is distinct from the resident device.
 4. The method of claim 1,wherein determining, by the resident device, whether the current stateof the accessory is different from the expected state of the accessoryfor the current context includes determining that the current contextcorresponds to an expected context and determining that the currentstate is different from the expected state.
 5. The method of claim 1,wherein, in accordance with a determination that the current state ofthe accessory is not different from the expected state of the accessoryfor the current context: determining whether the current context isdifferent from an expected context.
 6. The method of claim 1, whereinthe accessory state anomaly notification sent to the at least one userdevice comprises a graphical element configured to enable selection of areturn state element; and further comprising: receiving, from the atleast one user device, an indication of selection, at the at least oneuser device, of the return state element; and sending an instruction tothe accessory to return to the expected state for the current contextbased at least in part on the indication of the selection of the returnstate element.
 7. A non-transitory computer readable medium includingone or more sequences of instructions that, when executed by one or moreprocessors, cause: receiving, by a resident device, accessory event datathat indicates accessory state changes occurring at a plurality ofaccessory devices; based at least in part on the event data,identifying, by the resident device, a pattern of accessory statechanges that indicate an expected state for an accessory in acorresponding context; automatically generating, by the resident device,a plurality of scenes that include at least a subset of the plurality ofaccessory devices based at least in part on the pattern of accessorystate changes, wherein when the plurality of scenes are invoked, statesof the subset of the plurality of accessory devices are automaticallyadjusted; determining, by the resident device, a current state of theaccessory; determining, by the resident device, a current context forthe accessory; determining, by the resident device, whether the currentstate of the accessory is different from the expected state of theaccessory for the current context; and in accordance with adetermination that the current state of the accessory is different fromthe expected state of the accessory for the current context: generating,by the resident device, an accessory state anomaly notification thatindicates the current state of the accessory is unexpected for thecurrent context; determining, by the resident device, whether theaccessory and the current state of the accessory are part of a scene ofthe plurality of scenes, wherein the scene has been invoked at a time ofa day; in accordance with a determination that the accessory and thecurrent state of the accessory are not part of the scene: sending, bythe resident device, the accessory state anomaly notification to atleast one user device of a plurality of different user devices; and inaccordance with a determination that the accessory and the current stateof the accessory are part of the scene, wherein the resident devicerecognizes a pattern of the current state of the accessory anddetermines that the pattern is shifted from the time of the day to adifferent time: preventing, by the resident device, the accessory stateanomaly notification from being sent.
 8. The non-transitory computerreadable medium of claim 7, wherein the pattern of accessory statechanges includes an expected duration of the expected state in thecorresponding context; and wherein the instructions cause: determining,by the resident device, a current duration of the current state of theaccessory; determining, by the resident device, that the current stateof the accessory is different from the expected state of the accessoryfor the current context when the current duration of the currentaccessory state exceeds the expected duration of the expected state. 9.The non-transitory computer readable medium of claim 7, wherein theinstructions that cause receiving, by the resident device, the accessoryevent data include instructions that cause receiving the accessory eventdata from the at least one user device, where the at least one userdevice is distinct from the resident device.
 10. The non-transitorycomputer readable medium of claim 7, wherein the instructions that causedetermining, by the resident device, whether the current state of theaccessory is different from the expected state of the accessory for thecurrent context include instructions that cause determining that thecurrent context corresponds to an expected context and determining thatthe current state is different from the expected state.
 11. Thenon-transitory computer readable medium of claim 7, wherein, inaccordance with a determination that the current state of the accessoryis not different from the expected state of the accessory for thecurrent context: determining whether the current context is differentfrom an expected context.
 12. A system comprising: one or moreprocessors; and a non-transitory computer readable medium including oneor more sequences of instructions that, when executed by the one or moreprocessors, cause: receiving, by a resident device, accessory event datathat indicates accessory state changes occurring at a plurality ofaccessory devices; based at least in part on the event data,identifying, by the resident device, a pattern of accessory statechanges that indicate an expected state for an accessory in acorresponding context; automatically generating, by the resident device,a plurality of scenes that include at least a subset of the plurality ofaccessory devices based at least in part on the pattern of accessorystate changes, wherein when the plurality of scenes are invoked, statesof the subset of the plurality of accessory devices are automaticallyadjusted; determining, by the resident device, a current state of theaccessory; determining, by the resident device, a current context forthe accessory; determining, by the resident device, whether the currentstate of the accessory is different from the expected state of theaccessory for the current context; generating, by the resident device,an accessory state anomaly notification that indicates the current stateof the accessory is unexpected for the current context; determining, bythe resident device, whether the accessory and the current state of theaccessory are part of a scene of the plurality of scenes, wherein thescene has been invoked at a time of a day; in accordance with adetermination that the accessory and the current state of the accessoryare not part of the scene: sending, by the resident device, theaccessory state anomaly notification to at least one user device of aplurality of different user devices; and in accordance with adetermination that the accessory and the current state of the accessoryare part of the scene, wherein the resident device recognizes a patternof the current state of the accessory and determines that the pattern isshifted from the time of the day to a different time: preventing, by theresident device, the accessory state anomaly notification from beingsent.
 13. The system of claim 12, wherein the pattern of accessory statechanges includes an expected duration of the expected state in thecorresponding context; and wherein the instructions cause: determining,by the resident device, a current duration of the current state of theaccessory; determining, by the resident device, that the current stateof the accessory is different from the expected state of the accessoryfor the current context when the current duration of the currentaccessory state exceeds the expected duration of the expected state. 14.The system of claim 12, wherein the instructions that cause receiving,by the resident device, the accessory event data include instructionsthat cause receiving the accessory event data from the at least one userdevice, where the at least one user device is distinct from the residentdevice.
 15. The system of claim 12, wherein the instructions that causedetermining, by the resident device, whether the current state of theaccessory is different from the expected state of the accessory for thecurrent context include instructions that cause determining that thecurrent context corresponds to an expected context and determining thatthe current state is different from the expected state.
 16. The systemof claim 12, wherein, in accordance with a determination that thecurrent state of the accessory is not different from the expected stateof the accessory for the current context: determining whether thecurrent context is different from an expected context.